Doctoral Defense Application of Agent-Based Approaches to Enhance Container Terminal Operations by Omor Sharif Director: Dr. Nathan Huynh In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Civil Engineering College of Engineering and Computing Summer,

A Very Brief Review of Container Terminal An interface between ocean and land transport Containers are loaded/unloaded to/from a ship Temporary storage of export and import containers Operation involves a large number of decisions 2 Land Side Water Side Containe r Yard

Growth in Container Traffic, Major Challenges Limited capacity issues, Congestion Environmental concerns, emission from congestion Increased complexity in planning Lack of optimized decision making tools Competition among terminals has considerably increased Provide satisfactory customer service, reliability Tools that reduce cost, increase productivity of operations 3 Optimization | Automated Equipment | Information Technology World container volumes have increased from 113 million TEU to more than 572 million TEU in the period from 1993 to An annual average growth rate of nearly 10% per year.

Flow of Containers in a Container Terminal 4 Quay Cranes Internal Trucks Yard Crane s External Trucks External Trucks Customers Storage Area Berth Vessel Internal Truck

Decision Problems in Container Terminal  Berth Allocatio n  Quay Crane Scheduling Transpo rt Containers Storage Assignment Yard Crane Scheduling Gate Operations 5

Three Research Topics Research Focus Areas Queuing of Trucks at Gates Yard Crane Operations Container Storage Problem 6 Reduce truck processing time | Reduce congestion | Reduce emission Optimize available resources | Improve efficiency of related processes

Journal Article I Application of El Farol Model for Managing Marine Terminal Gate Congestion B by Omor Sharif, Nathan Huynh and Jose Vidal Research in Transportation Economics, 32 (1), (2011) 7

Excessive Waiting of Trucks at Terminal Gates Documentation processing, security checks at gate At peak hours demand greatly exceeds supply at gate Long waiting time results in high truck cost Source of significant emission, unhappy truckers 8 Figure: Port Elizabeth, New Jersey - Peak hour congestion at gate

How to Reduce Queuing Time? Experimental Solutions to Reduce Fluctuating Arrivals Appointment System with Time Windows Live View of Gate Using Webcams 9

Agent-based Model for Dispatch Decisions 10 Send Truck Now??? 4 miles

El Farol Bar Problem (Arthur, 1994) A bar in Santa Fe, New Mexico Every Thursday night, everyone wants to go to the El Farol But it's no fun to go there if, say, more than 60 go to the bar, they'll all have a worse time than if they stayed at home. It is necessary for everyone to decide at the same time whether they will go to the bar or not If agents use different strategies to decide, the overall attendance fluctuates at 60, which is the Nash Equilibrium 11 Fig: A simulated 100- week record of attendance at El Farol

Principal Parameters of Model 12 N ≡ Set of Depots (n ∈ N), T ≡ Set of Trucks (t ∈ T) Tolerance, L ≡ Maximum time the truck willing to wait Expected wait, E (W) ≡ The time truck thinks it will wait Make a Decision, SEND? (n, t) ≡ YESif E (W) ≤ L NO otherwise Wait at gate, W(t) = Waiting in Queue + Waiting in Service Discretization interval, I ≡ How often to make a decision? History x = Waiting time at terminal gate for last m intervals

Estimate E(W) using ‘predictors’ Predictors are simple rules or strategies/hypotheses Agents use predictive strategies to estimate E(W) Sample predictors-  A random number  Positive trend of last 10 values  A periodic function  Moving average of past 5 values 13

Solution Steps 14 Create: A large number of ‘predictors’ to estimate E(W) Choose: Agent chooses k predictors from predictor space, S Estimate: Estimate E(W) using each predictor Learn: Agents learns about how predictor has performed in past Rank: Agents assign scores and rank the predictors Use Best: Use ‘best performing predictor’ from predictors set

Model Implementation in Netlogo Simulation model, coded in Netlogo A modeling environment for agent-based systems Many useful primitives that expedite the implementation Extensive documentation and tutorials, model libraries Updated versions are continuously released 15

Experiment Parameters (Process 1440 Trucks) ParameterValueUnit Number of Depots10Nos Dispatch rate (θ) 12 trucks/depot/h r Mean transaction time (μ) 3, 4, 5, 6, 7 and 8 minutes Tolerance (L) 15, 20, 25 and 30 minutes Total predictors200Nos Predictors per depot (k) 12Nos / depot Update interval (I) 5,10 and 15minutes Maximum memory (m) 20intervals Predictor scoring policy Original precision n/a Alpha0.5n/a 16

Results (Impact of Tolerance, Interval on Wait) Fig: Impact of tolerance on mean wait time of trucks Fig: Impact of tolerance on max waiting time. 17 Increases fast at low I

Mean Wait Time Fluctuates Around Tolerance Fig: Mean wait time of trucks (I =15 minutes, L = 15 minutes) Fig: Mean wait time of trucks (I =10 minutes, L = 10 minutes) 18

Significantly Better Than ‘ Do Nothing’ 43% and 63% lower mean wait time for I = 5 and 10 mins 22% and 40% lower maximum wait time for I = 5 and 10 mins 19

Key Findings and Contributions Provides steady truck arrival, less queuing time at gate Adopt higher ‘I’ for distributing peak hour demand Good amount of emission (PM2.5, NOx, GHG) reduction First study on gate congestion using real-time information Examines how dispatchers can benefit from real-time information. Offers many experiments for choosing suitable parameters 20

Journal Article II Inter-Block Yard Crane Scheduling at a Marine Container Terminal by Omor Sharif, Nathan Huynh, Mashrur Chowdhury, Jose Vidal Transportation Science and Technology, 1 (2), (2012) 21

Efficiency of Container Yard Operations is Crucial Container yard is the interface of land and waterside operations Containers are temporarily stored in Container Yard A container yard is made up of several blocks of containers Workload varies (1) among blocks (2) over operational hours 22 Land/ Gate Side Water/ Vessel Side

Yard Cranes Influence Truck and Ship Waiting Time Yard Cranes are deployed for container handling Rubber Tired Gantry Cranes are most popular Yard Crane Cranes support the ship loading and unloading operations Cranes support external truck operations 23

Objective, Assumptions, Constraints Total operational hours is divided in several planning periods A workload forecast for blocks is known at start (time-units) At most two cranes work at a block at the same time At most one transfer per planning period 10 – 15 minutes transfer time for each block traveled 24 Given: A limited number of cranes and workload at blocks GOAL: Assign and relocate cranes among blocks to minimize the workload that remains unfinished at the end of a period

Rules for Initial Assignment of Cranes to Blocks  High to low work volume Sort blocks in decreasing order of initial work volume (IW b ) Assign NC b max cranes to each block starting from topmost item  Crane at each block If n c = n b, assign one crane per block If n c < n b, sort blocks in decreasing IW b, assign cranes If n c > n b, first assign one crane per block, then sort in decreasing IW b  Reduce transfers Assign cranes to reduce number of future transfers A four step approach Based on pass/fail of a series of inequalites 25

Three Principal Parameters of Model  Extra capacity of a crane E (c) = T c × NC b initial − IW b Time Period X Initial Number of Cranes – Workload of block  Amount of help needed by a block H (b) = IW b − T c × NC b current Workload of block - Time Period X Current Number of Cranes  Transfer time matrix TT c od Different transfer time for longitudinal and lateral movement 26

Matching using Deferred Acceptance Algorithm Assigning cranes are viewed as matching crane and blocks DAA is a market matching model by Gale and Shapley Numerous follow up studies in Economics, Computer science Find a match between two set of agents Each agent has a preference order over other set of agents ‘One to one’ matching – e.g. Marriage Problem ‘Many to one’ matching – e.g. College Admission 27

Preference Functions For Crane And Block Agents Minimum transfer time Transfer time of a crane from origin to destination block Positive difference Extra Capacity of a crane – Help needed by a block – Transfer time Absolute difference abs (Extra Capacity of a crane – Help needed by a block – Transfer time) Absolute difference squared distance abs (Extra Capacity of a crane – Help needed by a block – Transfer time 2 ) 28

Algorithm to assign cranes to blocks Each crane j send match request to first block i from its preference list 2. Each block i receiving more than q i requests, ‘holds’ the most preferred q i cranes and rejects all others. n. Each crane j rejected at step n − 1 removes the block i rejecting the crane from its preference list. Then the rejected crane j makes a new match request to its next most preferred block i who hasn’t yet rejected it. Go to step n − 1. (n = 3, 4, 5, …)

Implementation Snapshot from Netlogo Multi-agent simulation GUI 30 Setup block layout Create workload Create cranes Initial assignment Inter-block transfer matrix Pre-analysis steps Find Matching Reassign/update results

Test Parameters Based on Real-World Problems ParameterValueUnit Number of blocks10, 20, 30Nos Number of cranes1 or 1.5 × Number of blocksNos Length of planning period240Minutes Work volume (1) Moderate (2) Heavy (3) Above capacity Minutes Initial assignment (1) Crane at each block (2) High to low work volume (3) Reduce transfers - Preference strategy (1) Minimum transfer time (2) Absolute difference (3) Positive difference (4) Absolute inverse squared distance - Algorithm Version(1) Blocks proposing (2) Cranes proposing - 31 (1)Moderate Workload: 60% of total crane capacity ± 40% of average workload (2)Heavy Workload: 90% of total crane capacity ± 20% of average workload (3)Above Capacity Workload: 110% of total crane capacity ± 40% of average workload

Model Effectively Reduces Unfinished Workload Percentage incomplete work volume: (Incomplete/Total*100%) Case I - Average number of cranes per block = 1.0 Number of Blocks Number of Cranes Work VolumeMH ACACMH ACACMH ACAC Minimum Transfer Time Absolute difference Positive difference Absolute difference squared distance Mathematical Program

Majority of Cases are Optimal/Near-optimal Percentage incomplete work volume: Case II - Average number of cranes per block = 1.5 Number of Blocks Number of Cranes Work VolumeMHACACMHACACMHACAC Minimum Transfer Time Absolute difference Positive difference Absolute difference squared distance Mathematical Program

Key Findings and Contributions In ‘medium’ condition all work can be finished In ‘heavy’ condition the percentage incomplete is 1% or less In ‘above capacity’ condition the percentage remaining is within 3% of the lower bound. ‘Reduce Transfers’ initial assignment works best. An agent-based approach for solving inter-block scheduling problem Provide several preference functions that are effective in minimizing Scalable and time efficient, a problem with 30 blocks can be solved in 3 seconds Several minutes to solve using Mixed Integer Programs Provides various strategies for initial assignment of cranes to blocks 34

Journal Article III Storage Space Allocation at Marine Container Terminals Using Ant-based Control by Omor Sharif and Nathan Huynh Expert Systems with Applications, 40 (6), (2013) 35

Assign Newly Arriving Containers to Yard Blocks Container yard is divided up into yard blocks Each yard block has a fixed total capacity Store export containers brought in by trucks Store import containers unloaded from ships Allocate yard blocks to newly arriving containers 36 Gates Berths Yard Blocks Export containers Import containers Export containers Import containers Land Side Container Yard Water Side

Two Competing Objectives of SSAP Balance the workload of blocks– Proper distribution of arriving containers amongst blocks Minimize container transportation cost – Minimize the distance traveled by ITs between berth and yard 37

Reduces relocation of yard cranes Reduces waiting time of vessels Reduces average container handling time Reduces congestion on road network Improves efficiency of related resources (quay cranes, yard cranes, storage space, ITs) – Zhang et al., 2003 Workload Balancing is Critical Because… 38 Workload Imbalance is the variability of number of containers among blocks at some time (we used standard deviation for this measure).

We Utilize an Ant-Based Control ABC Falls under ‘Ant colony optimizations & algorithms’ Inspired by collective intelligence of an ant colony Can solve complex routing problems in network For example, used for shortest path, load balancing We utilize pheromone laying behavior of ants Ants have a decision to make Situation several months later 39

Highlights of Ant Based Control for SSAP A container terminal is a network of gates, blocks, berths Links are bi-directional container transportation routes Network also support a traffic of artificial ant agents Ants move just like inbound & outbound containers Ants lay pheromone as a function of distance & congestion Probabilities in pheromone table guide ants & containers Fig: Ant choses path according to probability 40

Shortest Path Workload Balancing Ant age Increases with traveled distance Probability of selection decrease with ant age Ants are delayed at congested blocks Delay is proportional to degree of congestion 41

Dynamic Interaction between Ants & Containers 42

Simulation The snapshot of GUI of our simulation 43

Parameter and Values Used in Model 44 A demand of 2500 Containers

Parameter and Values Used in Model 45

Performance Metric I - Transport Distance Converge to shortest path in 500 time-units Under no congestion converge to lower bound When congestion occurs distance converge to 2% above lower bound 46

Performance Metric II - Workload Imbalance Standard deviation gradually rises when no congestion When congestion occurs standard deviation falls Maintains a distribution to accommodate both objectives 47

Key Findings and Contributions The approach is effective in balancing workload in blocks Reduces transport distances during ship loading/unloading Distributed allocation to individual containers Real-time solution requiring no advanced container arrival information, more suitable for real-world truck arrival Adaptive, scalable. Synchronous instead of traditional hierarchical approach 48

Thank You 49

A class of computational modeling For simulating the actions and interactions of autonomous, intelligent agents To assess the agents’ effects on the system as a whole. ABMs are individual-based models Most agent-based models are often composed of: (1) agents with simple or complex goals (2) distributed decision-making heuristics; (3) learning rules or adaptive processes; (4) an interaction system; and (5) a non-agent environment What is Multi-agent Systems and Agent-Based Modeling? 50

ABM is relatively new research field within artificial intelligence. ABM is relatively unexplored in the area of terminal operations. Suitable for dynamic distributed resource allocation Traditional approaches use centralized mathematical programs for optimization No Polynomial time solution for majority of problems ABM can outperform in certain domain property Scalability: Do not require exhaustive search in solution space, model integration potential Computational Efficiency: Real time solution, no advanced computation time is required Adaptability: No forecast required, no revision for forecast errors Fault tolerance: Failure will impact locally, thus more robust Why use agent-based modeling in a terminal? 51

In Recent years integrative modeling for container terminals are being emphasized Various processes in a terminal are interconnected Improved terminal performance cannot necessarily be achieved by treating the processes separately A few studies have attempted such an integrated approach, and they are on the quay side More decision tools need to be developed and integrated Multiagent systems approach has been proposed as a tool in integrated decisions frameworks in works by Henesey (2006); Franz et al. (2007), Stahlbock and VoB (2008) Integrated model using traditional optimization techniques has limited viability due to computational complexity. Integrative Modeling in CT Research 52

Drayage is relatively short distance of supply chain However, drayage accounts for 25% to 40% of Origin- Destination Expenses High cost affect the profitability of an intermodal service Reduce the emissions impact on the surrounding communities Reducing the idling time of drayage trucks is equivalent to reducing local and regional particulate matter (PM 2.5), nitrogen oxides, and greenhouse gas emissions Drayage trucks operate primarily in urban environments, a reduction of these harmful pollutants has a proportionally greater benefit. Why study drayage? 53

Terminal operator designates available time windows, truckers choose one of the available time windows Experimented in terminals e.g. Los Angeles, Long Beach, Oakland, Port of Vancouver etc. Under consideration at others. In theory, the appointment system should reduce truck processing time In practice, little time savings for truckers was reported due to lack of structures in appointment system Appointment systems are in an early stage of development, with no uniformity between terminals or ports and many implementation issues to be resolved Appointment System is at early stage of development 54

The problem addressed in this study is more complex than the original El Farol Bar First, the total demand (number of people intending to go to the bar) does not vary with time, truck demands vary throughout the day. Second, the bar attendance event occurs at one specific time. Truck dispatching a continuous process that occurs throughout the operational hours of the terminal. Third, depots are not homogenous, they are located at different distances; depots closer to the terminal are in a better position to take advantage of the provided real- time information, E(W) will be closer to the actual wait time. Fourth, travel time and service time are not applicable to bar problem. Given service time is stochastic, the strategy of estimating truck wait time is more difficult. Additional difficulty of Decision making by a truck dispatcher VS Individuals in the bar 55

For each block the initial work volume at the beginning of a planning period is the work volume forecast for that period plus incomplete work volume from the previous period. Work volume = Average timeunits required per move X Number of container moves How to Determine Workload in a Yard Block 56

Proposed Agent-based Model (Contd.) Pseudo Code of the Program – Part of the Main Loop 57

A Sample Example of Interblock YC Schedule 58 Initial Assignment Obtained via ‘Reduce Transfer’ (block 0, crane 11); (block 3, crane 19); (block 2, crane 18); (block 7, crane 16); (block 5, crane 14) Mathematical program gives the same assignments

A sample example (Contd.) 59

Some of the preferences in this example are symmetric The crane’s first choice is a block whose first choice is the crane (e.g. Block 0 and Crane 11; Block 7 and Crane 16; Crane 14 and Block 5). Notice that Block 3 and Block 0 both wants Crane 11; Block 0 gets it because it is the first item of Crane 11’s list; Block 3 ends up getting 4th choice) We can pick different preference strategies; for example, the cranes may use ‘minimum transfer time’, whereas the blocks may use ‘absolute difference’. Additional Notes on the Sample Example 60

Overview of Storage Space Allocation Problem (SSAP) 61

 Block level decision- Finds the block to store a container Objectives are ‘workload balancing’ & ‘distance minimizing’  Stack level decision- Finds an exact stack in a block Objective is to ‘minimize future reshuffling’ Two Levels of SSAP 62

63