DriveLM的baseline复现

DriveLM是一篇很有意思的工作，把自动驾驶跟MLLM结合到一起了，实现端到端的感知or决策规划。

Repo：https://github.com/OpenDriveLab/DriveLM

该工作是基于nuScenes数据集做的，官方paper里给出了数据的具体构建方式，感兴趣的可以移步https://arxiv.org/abs/2312.14150

官方也给出了榜单，本文章主要复现该榜单的baseline，从而熟悉一下相关工作。

首先下载Repo，然后cd到challenge/目录，相关的baseline训练评估测试都在这个目录下进行。

官方给出的baseline以 LLaMA-7B和 llama-adapter v2 实现。

一、环境配置

按照官方文档配置，我没有进行修改，用的CUDA11.8+A100显卡

二、权重准备

准备好LLaMA-7B（注意不是LLaMA-2-7B）的相关权重文件，可以通过https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform?usp=send_form申请得到下载的url，因为我申请没被通过（不知道为啥。。用了学校的edu邮箱），因此去hf上找了一个别人上传的权重仓库https://huggingface.co/nyanko7/LLaMA-7B/tree/main。

下载在本地后按照下图的结构布置整个文件目录

此外，从https://github.com/OpenGVLab/LLaMA-Adapter/releases/tag/v.2.0.0下llama_adapter_v2的相关权重（以下所有pre-trained checkpoints或者ckpt都指这个llama_adapter_v2而非LLaMA-7B的相关权重），目前有三个权重文件，选哪个都差不多。

三、数据准备（demo）

readme里的数据写的有点混乱，我整理了关于原始仓库附带的数据/json文件。在不下载任何额外数据和操作的情况下，利用原始仓库自带的test_llama.json就可以跑一个infer的demo

目录	备注
challenge/data/train_sample.json	训练数据的demo，对应该json相关的图片等数据在challenge/llama_adapter_v2_multimodal7b/data/nuscenes/samples
challenge/test_llama.json	用于baseline（llama-7B）的train和infer的输入demo数据
challenge/output.json	用于baseline（llama-7B）的infer输出的demo数据
https://huggingface.co/datasets/OpenDriveLab/DriveLM/blob/main/v1_1_train_nus.json	DriveLM完整的annotation数据:train
https://huggingface.co/datasets/OpenDriveLab/DriveLM/blob/main/v1_1_val_nus_q_only.json	DriveLM完整的annotation数据:val
https://huggingface.co/datasets/OpenDriveLab/DriveLM/blob/main/drivelm_nus_imgs_train.zip& https://huggingface.co/datasets/OpenDriveLab/DriveLM/blob/main/drivelm_nus_imgs_val.zip	DriveLM完整的数据:train和val

四、推理demo

准备完毕后，通过下方一行代码实现zero-shot的推理infer demo

# /path/to/llama_model_weights and /path/to/pre-trained/checkpoint.pth need to be modified by your path
# num_processes is the number of the gpu you will use to infer the data.
# make sure you are under ./challenge/llama_adapter_v2_multimodal7b
## 注意这里的checkpoint是任意一个llama-adapter-v2的权重
## --data对应的是原始仓库的 test_llama.json路径 --output是保存的推理答案路径
## --num_processes是GPU数目, 单卡的话改成1
python demo.py --llama_dir /path/to/llama_model_weights --checkpoint /path/to/pre-trained/checkpoint.pth --data ../test_llama.json  --output ../output.json --batch_size 4 --num_processes 8

推理速度大概12s/batch（A100 bs=4）得到下方对应test_llama.json中17段数据推理的答案，如果answer是什么\n \n乱七八糟的，大概率是权重不对，代码和数据没问题。我这边用的是llama-adapter-v2的权重中的7fa55208379faf2dd862565284101b0e4a2a72114d6490a95e432cf9d9b6c813_BIAS-7B.pth

实测这个推理结果跟官方给的output.json答案一致，如果用其他ckpt，答案会不同，但是不同的答案也算合理，不会出现乱码。

[{"id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_0","question": "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.","answer": "In the current scene, there are several important objects that need to be considered for the future reasoning and driving decision. These objects include a car driving down the street, a traffic light, a stop sign, a street sign, and a building. The car's position and speed, as well as the traffic light's status, are crucial factors in determining the driver's next move. The stop sign and street sign provide information about the street's name and any potential restrictions or directions. The building in the background adds context to the scene, providing a sense of the surroundings and the environment in which the car is driving."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_1","question": "<image>\nWhat is the moving status of object <c1,CAM_BACK,1088.3,497.5>? Please select the correct answer from the following options: A. Going ahead. B. Stopped. C. Back up. D. Turn left.","answer": "The moving status of object <c1,CAM_BACK,1088.3,497.5> is \"Back up.\""},{"id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_2","question": "<image>\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?","answer": "1. The ego vehicle should notice the traffic light first when getting to the next possible location. The traffic light is in the middle of the road, and it is important for the ego vehicle to pay attention to it to ensure safe navigation.2. The state of the traffic light is red, indicating that the ego vehicle should stop at the intersection. The ego vehicle should come to a complete stop and wait for the traffic light to change to green before proceeding.3. The ego vehicle should notice the traffic light second when getting to the next possible location. The traffic light is still in the middle of the road, and it is important for the ego vehicle to pay attention to it to ensure safe navigation.4. The state of the traffic light is green, indicating that the ego vehicle can proceed through the intersection. The ego vehicle should continue driving through the intersection, following the traffic rules and guidelines."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_3","question": "<image>\nIs <c1,CAM_BACK,1088.3,497.5> a traffic sign or a road barrier?","answer": "The traffic sign is located on the side of the road, and it is not a road barrier."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_4","question": "<image>\nWhat actions could the ego vehicle take based on <c1,CAM_BACK,1088.3,497.5>? Why take this action and what's the probability?","answer": "Based on the given coordinates, the ego vehicle could take the action of turning left at the intersection. The probability of this action is high, as the vehicle is already at the intersection and has the right-of-way. Turning left at the intersection is a common action for vehicles to take when they are approaching an intersection and need to make a left turn. The probability of this action is high because the vehicle is already at the intersection and has the right-of-way, which means it is more likely to turn left than to continue straight or turn right."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_5","question": "<image>\nWhat actions taken by the ego vehicle can lead to a collision with <c1,CAM_BACK,1088.3,497.5>?","answer": "The ego vehicle can lead to a collision with <c1,CAM_BACK,1088.3,497.5> by not paying attention to the traffic light and failing to stop at the red light. The vehicle is driving through the intersection, which is a dangerous situation as it could collide with other vehicles or pedestrians. The driver should have stopped at the red light and waited for it to turn green before proceeding."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_6","question": "<image>\nIn this scenario, what are safe actions to take for the ego vehicle?","answer": "In the image, the ego vehicle is driving through a rainy city street with a traffic light. To ensure safety, the driver should follow these actions:1. Maintain a safe distance from other vehicles: Keep a safe distance from other vehicles, especially if visibility is low due to the rain.2. Reduce speed: Slow down to allow for better visibility and to avoid hydroplaning.3. Use headlights: Turn on headlights to increase visibility for both the driver and other road users.4. Be cautious at intersections: Be extra vigilant at intersections, as visibility may be reduced, and other drivers may not be able to see you clearly.5. Be prepared for sudden stops: Anticipate the need to stop suddenly due to wet roads, and be prepared to react quickly.6. Avoid distractions: Avoid using cell phones, eating, or engaging in other activities that may distract the driver from the road conditions.By following these safety measures, the ego vehicle driver can ensure a safer and more comfortable driving experience in the rainy city street."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_7","question": "<image>\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving fast. B. The ego vehicle is going straight. The ego vehicle is driving fast. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving with normal speed.","answer": "The ego vehicle is slightly steering to the left, and it is driving with normal speed."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_0","question": "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.","answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a truck, a bus, a car, and a building. The truck and bus are parked on the street, while the car is driving down the road. The building is located on the right side of the street, and there are also some trees in the background. The presence of these objects and their positions in the scene will be taken into account when making future driving decisions, such as navigating around the parked vehicles, avoiding potential collisions, and ensuring safe driving conditions for all road users."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_1","question": "<image>\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?","answer": "1. The ego vehicle should notice the traffic light first, as it is the first object in the image. The traffic light is in the process of changing from red to green, indicating that the vehicle should proceed with caution.2. The ego vehicle should notice the truck next, as it is the second object in the image. The truck is stopped at the intersection, and the ego vehicle should be cautious when approaching the intersection to ensure it does not collide with the truck.3. The ego vehicle should notice the bus next, as it is the third object in the image. The bus is stopped at the intersection, and the ego vehicle should be cautious when approaching the intersection to ensure it does not collide with the bus.4. The ego vehicle should notice the traffic light next, as it is the fourth object in the image. The traffic light is now green, indicating that the vehicle can proceed through the intersection.5. The ego vehicle should notice the truck next, as it is the fifth object in the image. The truck is now moving, and the ego vehicle should be cautious when approaching the intersection to ensure it does not collide with the truck."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_2","question": "<image>\nIs <c1,CAM_BACK,991.7,603.0> a traffic sign or a road barrier?","answer": "The image shows a traffic sign, not a road barrier."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_3","question": "<image>\nWhat actions could the ego vehicle take based on <c1,CAM_BACK,991.7,603.0>? Why take this action and what's the probability?","answer": "Based on the given coordinates, the ego vehicle could take the action of turning left at the intersection. The probability of this action is high, as the vehicle is already at the intersection and has the right-of-way. Turning left would allow the vehicle to continue its journey on the road, avoiding any potential collisions with other vehicles or pedestrians."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_4","question": "<image>\nWhat actions taken by the ego vehicle can lead to a collision with <c1,CAM_BACK,991.7,603.0>?","answer": "The ego vehicle can lead to a collision with <c1,CAM_BACK,991.7,603.0> by not paying attention to the traffic light and failing to stop at the red light. The ego vehicle is driving through the intersection, and if it does not stop at the red light, it could collide with the other vehicle. This is a common cause of accidents at intersections, as drivers often fail to follow traffic rules and regulations."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_5","question": "<image>\nIn this scenario, what are safe actions to take for the ego vehicle?","answer": "In the image, the ego vehicle is driving through a rainy city street. To ensure safety, the driver should take the following actions:1. Maintain a safe distance from other vehicles: In wet conditions, the stopping distance increases, so it is essential to leave a safe distance between the ego vehicle and other vehicles on the road.2. Reduce speed: Driving at a slower speed helps to maintain better control of the vehicle and allows for better reaction time in case of sudden changes in road conditions.3. Use headlights: Turn on the headlights to increase visibility for both the driver and other road users.4. Be cautious of pedestrians: Rainy weather can make it more difficult for pedestrians to see and be seen, so the driver should be extra vigilant and maintain a safe speed when approaching intersections or crosswalks.5. Avoid sudden braking or acceleration: Sudden changes in speed or direction can lead to loss of control and increased risk of accidents.6. Be prepared for hydroplaning: If the vehicle starts to hydroplane, the driver should gently apply the brakes and steer in the direction of the skid to regain control"},{"id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_6","question": "<image>\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving very fast. B. The ego vehicle is steering to the left. The ego vehicle is driving fast. C. The ego vehicle is steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving slowly.","answer": "The ego vehicle is driving with normal speed and is steering to the left."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_0","question": "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.","answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a truck, a bus, a car, a traffic light, and a street sign. The presence of these objects indicates that the scene is likely a busy intersection or a city street, where vehicles are moving and interacting with each other. The traffic light and street sign provide essential information for drivers to navigate the area safely and efficiently. The truck and bus are also important, as they are large vehicles that require extra attention and care when sharing the road. The car, as the smallest vehicle in the scene, may also be considered as a potential hazard or obstacle for the other vehicles. Overall, these objects will be crucial for drivers to understand the context and make informed decisions while navigating the busy street."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_1","question": "<image>\nWhat is the moving status of object <c1,CAM_FRONT_LEFT,231.5,472.1>? Please select the correct answer from the following options: A. Stopped. B. Going ahead. C. Turn right. D. Turn left.","answer": "The moving status of object <c1,CAM_FRONT_LEFT,231.5,472.1> is \"Stopped.\""},{"id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_2","question": "<image>\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?","answer": "1. The ego vehicle should notice the traffic light first, as it is the first object in the image. The traffic light is in the process of changing from red to green, indicating that the vehicle should proceed with caution.2. The ego vehicle should notice the truck next, as it is the second object in the image. The truck is stopped at the intersection, and the ego vehicle should be cautious and wait for the truck to move before proceeding.3. The ego vehicle should notice the traffic light again, as it is the third object in the image. The traffic light is now green, indicating that the vehicle can proceed.The ego vehicle should take the appropriate action based on the state of the traffic light and the position of the other vehicles on the road. If the traffic light is red, the ego vehicle should stop and wait for the light to change. If the traffic light is green, the ego vehicle should proceed with caution, being mindful of other vehicles and pedestrians in the area."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_3","question": "<image>\nIs <c1,CAM_FRONT_LEFT,231.5,472.1> a traffic sign or a road barrier?","answer": "The traffic sign is located on the side of the road, while the road barrier is located in the middle of the road."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_4","question": "<image>\nWhat actions could the ego vehicle take based on <c1,CAM_FRONT_LEFT,231.5,472.1>? Why take this action and what's the probability?","answer": "Based on the given coordinates, the ego vehicle could take the action of turning left at the intersection. The probability of this action is high, as the vehicle is already at the intersection and has the right-of-way. Turning left would allow the vehicle to continue its journey on the road, avoiding any potential conflicts with other vehicles or pedestrians."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_5","question": "<image>\nWhat actions taken by the ego vehicle can lead to a collision with <c1,CAM_FRONT_LEFT,231.5,472.1>?","answer": "The ego vehicle can lead to a collision with <c1,CAM_FRONT_LEFT,231.5,472.1> by not following the traffic rules or driving in an unsafe manner. Some actions that can lead to a collision include:1. Failing to stop at a red light or stop sign, as seen in the image.2. Driving too fast for the current road conditions, such as driving through a puddle or rain.3. Failing to maintain a safe distance from other vehicles, especially when approaching a stop sign or intersection.4. Failing to yield the right of way to pedestrians or other vehicles at a crosswalk or intersection.5. Driving distracted or under the influence of alcohol or drugs.By taking these actions, the ego vehicle can put itself and other road users at risk of a collision. It is essential for drivers to be aware of their surroundings, follow traffic rules, and drive responsibly to ensure the safety of themselves and others on the road."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_6","question": "<image>\nIn this scenario, what are safe actions to take for the ego vehicle?","answer": "In the image, the ego vehicle is driving down a wet street, which can be a challenging situation due to the wet road conditions. To ensure safety, the driver should take the following actions:1. Slow down: Driving at a slower speed helps maintain better control of the vehicle and reduces the risk of skidding or hydroplaning.2. Keep a safe distance: Maintain a safe distance from other vehicles and obstacles, as the wet road conditions can make it more difficult to stop in time.3. Use appropriate tires: Ensure that the vehicle is equipped with appropriate tires for wet conditions, such as all-season or winter tires, which have better traction and grip on wet surfaces.4. Be cautious of pedestrians: Be extra vigilant for pedestrians, as they may be more difficult to see in the rain.5. Avoid sudden braking or acceleration: Sudden changes in speed or direction can cause the vehicle to lose traction, leading to a loss of control.By following these safety measures, the ego vehicle driver can minimize the risk of accidents and ensure a safe journey on the wet street."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_7","question": "<image>\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving very fast. B. The ego vehicle is going straight. The ego vehicle is driving slowly. C. The ego vehicle is going straight. The ego vehicle is not moving. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving with normal speed.","answer": "The ego vehicle is slightly steering to the left, and it is driving with normal speed."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_0","question": "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.","answer": "In the current scene, there are several important objects that need to be considered for the future reasoning and driving decision. These objects include a car, a truck, a bus, a street sign, and a traffic light. The presence of these objects indicates that the scene is a busy urban area with multiple vehicles and pedestrians. The traffic light is an essential element for controlling the flow of traffic and ensuring safety. The street sign provides information about the location and direction of the street, which is crucial for navigation and understanding the context of the scene. The presence of a truck and a bus indicates that the area is likely a commercial or transportation hub, with vehicles carrying goods or passengers. The car is a common sight in urban areas, and its presence indicates that the scene is a typical urban environment with various modes of transportation. The combination of these objects creates a complex and dynamic environment that drivers need to navigate and adapt to."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_1","question": "<image>\nWhat is the moving status of object <c1,CAM_BACK,1568.3,505.0>? Please select the correct answer from the following options: A. Stopped. B. Turn left. C. Going ahead. D. Turn right.","answer": "The moving status of object <c1,CAM_BACK,1568.3,505.0> is \"Going ahead.\""},{"id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_2","question": "<image>\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?","answer": "1. The ego vehicle should notice the traffic light first when getting to the next possible location. The traffic light is in the foreground of the image, and it is the first object that the ego vehicle should notice.2. The state of the traffic light is red, indicating that the ego vehicle should stop at the intersection. The ego vehicle should come to a complete stop at the red light, as per traffic rules and regulations.3. The ego vehicle should notice the street sign next. The street sign is located behind the traffic light, and it is the second object that the ego vehicle should notice.4. The state of the street sign is unclear, as it is partially obscured by the traffic light. The ego vehicle should pay attention to the street sign and ensure that it is visible before proceeding.5. The ego vehicle should notice the car in the background. The car is located behind the street sign, and it is the third object that the ego vehicle should notice.6. The state of the car in the background is unclear, as it is partially obscured by the street sign. The ego vehicle should pay attention to the car in the background and ensure that it is visible before proceeding."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_3","question": "<image>\nIs <c1,CAM_BACK,1568.3,505.0> a traffic sign or a road barrier?","answer": "The image shows a traffic sign, not a road barrier."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_4","question": "<image>\nWhat actions could the ego vehicle take based on <c1,CAM_BACK,1568.3,505.0>? Why take this action and what's the probability?","answer": "Based on the given coordinates, the ego vehicle could take the action of turning left at the intersection. The probability of this action is high, as the vehicle is already at the intersection and has the right-of-way. Turning left would allow the vehicle to continue its journey on the road, avoiding any potential collisions with other vehicles or pedestrians."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_5","question": "<image>\nWhat actions taken by the ego vehicle can lead to a collision with <c1,CAM_BACK,1568.3,505.0>?","answer": "The ego vehicle can lead to a collision with <c1,CAM_BACK,1568.3,505.0> by not paying attention to the traffic situation and failing to maintain a safe distance from the other vehicle. The ego vehicle should be aware of the other vehicle's position and speed, and adjust its speed and course accordingly to avoid a collision. This includes maintaining a safe following distance, slowing down or stopping if necessary, and being prepared to react to sudden changes in the other vehicle's behavior."},{"id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_6","question": "<image>\nIn this scenario, what are safe actions to take for the ego vehicle?","answer": "In the image, the ego vehicle is driving through a rainy city street. To ensure safety, the driver should take the following actions:1. Maintain a safe distance from other vehicles: The wet road conditions can make it more difficult to stop or maneuver, so it is essential to maintain a safe distance from other vehicles.2. Reduce speed: Driving at a slower speed helps to maintain better control of the vehicle and allows for more time to react to any sudden changes in the road or traffic conditions.3. Use headlights: Turn on the headlights to increase visibility for both the driver and other road users.4. Be cautious of pedestrians: The wet road conditions may make it more difficult for pedestrians to maintain their footing, so the driver should be extra vigilant and slow down when approaching crosswalks or intersections.5. Avoid sudden braking or acceleration: Sudden changes in speed or direction can cause the vehicle to lose traction, leading to skidding or hydroplaning.6. Be prepared for puddles: The driver should be prepared for puddles and other wet spots on the road, as they may cause"},{"id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_7","question": "<image>\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. B. The ego vehicle is slightly steering to the right. The ego vehicle is driving very fast. C. The ego vehicle is going straight. The ego vehicle is driving fast. D. The ego vehicle is steering to the left. The ego vehicle is driving with normal speed.","answer": "The ego vehicle is slightly steering to the left, and it is driving slowly."},{"id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_0","question": "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.","answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a traffic light, a street sign, a building, and a car. The traffic light is located at the intersection, and it is important for drivers to pay attention to its position and color to ensure they follow the traffic rules and regulations. The street sign provides information about the street name, which is essential for drivers to navigate the area. The building is situated near the intersection, and it could be a landmark or a business establishment, which drivers should be aware of when navigating the area. The car is parked on the side of the road, and it might be a potential obstacle for drivers or pedestrians. Overall, these objects play a crucial role in the decision-making process of drivers and pedestrians in the area, and they should be considered while navigating the street."},{"id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_1","question": "<image>\nWhat is the moving status of object <c1,CAM_FRONT,714.3,503.6>? Please select the correct answer from the following options: A. Stopped. B. Back up. C. Turn left. D. Going ahead.","answer": "The moving status of object <c1,CAM_FRONT,714.3,503.6> is \"Stopped.\""},{"id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_2","question": "<image>\nIs <c1,CAM_FRONT,714.3,503.6> a traffic sign or a road barrier?","answer": "The image description does not provide enough information to definitively determine whether the object is a traffic sign or a road barrier. However, it is possible that it could be either one, depending on the context and the specific details of the object."},{"id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_3","question": "<image>\nWhat actions could the ego vehicle take based on <c1,CAM_FRONT,714.3,503.6>? Why take this action and what's the probability?","answer": "Based on the given coordinates, the ego vehicle could take the action of turning left at the intersection. The probability of this action is high, as the vehicle is already at the intersection and has the right-of-way to turn left. The vehicle is likely to turn left to continue its journey, as it is already in the intersection and has the right-of-way."},{"id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_4","question": "<image>\nWhat actions taken by the ego vehicle can lead to a collision with <c1,CAM_FRONT,714.3,503.6>?","answer": "The ego vehicle can lead to a collision with <c1,CAM_FRONT,714.3,503.6> by:1. Failing to stop at the red traffic light: If the ego vehicle fails to stop at the red traffic light, it may collide with the other vehicle that is already stopped at the intersection.2. Failing to yield to the other vehicle: If the ego vehicle fails to yield to the other vehicle that is already stopped at the intersection, it may collide with the vehicle that is waiting to turn left.3. Turning left without checking for oncoming traffic: If the ego vehicle turns left without checking for oncoming traffic, it may collide with the other vehicle that is already stopped at the intersection.4. Turning left without checking for pedestrians: If the ego vehicle turns left without checking for pedestrians, it may collide with pedestrians crossing the street.In summary, the ego vehicle can lead to a collision with <c1,CAM_FRONT,714.3,503.6> by failing to stop at the red traffic light, failing to yield to the"},{"id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_5","question": "<image>\nIn this scenario, what are safe actions to take for the ego vehicle?","answer": "In the image, the ego vehicle is driving down a street with a traffic light at the intersection. To ensure safety, the ego vehicle should follow the traffic light's instructions and obey the traffic rules. This includes stopping at red lights, yielding to pedestrians and other vehicles, and maintaining a safe distance from other vehicles. Additionally, the driver should be aware of their surroundings, including pedestrians, cyclists, and other vehicles, and be prepared to react to any unexpected situations. By following these safety measures, the ego vehicle can help prevent accidents and maintain a smooth flow of traffic in the area."},{"id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_6","question": "<image>\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. C. The ego vehicle is going straight. The ego vehicle is driving slowly. D. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed.","answer": "The ego vehicle is slightly steering to the left, and it is driving with normal speed."},{"id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_0","question": "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.","answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a traffic light, a car, a building, and a street sign. The traffic light is located at the intersection, and it is important for drivers to pay attention to its position and color to determine whether they can proceed or need to stop. The car is driving on the road, and it is essential for the driver to be aware of its position and speed to avoid collisions or accidents. The building is situated near the intersection, and it may be a landmark or a point of interest for the driver. The street sign provides information about the street name, which can be helpful for navigation or identifying the location. Overall, these objects play a crucial role in the driver's decision-making process and contribute to the overall safety and efficiency of the road system."},{"id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_1","question": "<image>\nWhat is the moving status of object <c1,CAM_FRONT,767.5,513.3>? Please select the correct answer from the following options: A. Reverse parking. B. Back up. C. Turn right. D. Stopped.","answer": "The moving status of object <c1,CAM_FRONT,767.5,513.3> is \"Stopped.\""},{"id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_2","question": "<image>\nIs <c1,CAM_FRONT,767.5,513.3> a traffic sign or a road barrier?","answer": "The traffic sign is located on the side of the road, and it is not a road barrier."},{"id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_3","question": "<image>\nWhat actions could the ego vehicle take based on <c1,CAM_FRONT,767.5,513.3>? Why take this action and what's the probability?","answer": "Based on the given coordinates, the ego vehicle is located at the intersection of two streets, with a traffic light in front of it. The ego vehicle could take several actions, such as:1. Continue straight: The ego vehicle could continue driving straight through the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and the traffic light is indicating that it should proceed.2. Turn left: The ego vehicle could turn left at the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and the traffic light is indicating that it should proceed.3. Turn right: The ego vehicle could turn right at the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and the traffic light is indicating that it should proceed.4. Stop: The ego vehicle could stop at the intersection, waiting for the traffic light to change or for other vehicles to pass before proceeding. This action has a probability of 0% as the vehicle is already at the intersection and the traffic"},{"id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_4","question": "<image>\nWhat actions taken by the ego vehicle can lead to a collision with <c1,CAM_FRONT,767.5,513.3>?","answer": "The ego vehicle can lead to a collision with <c1,CAM_FRONT,767.5,513.3> by failing to yield to the traffic light, which is red, and proceeding through the intersection. This can result in a collision with the other vehicle, which is waiting at the intersection."},{"id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_5","question": "<image>\nIn this scenario, what are safe actions to take for the ego vehicle?","answer": "In the image, the ego vehicle is approaching a traffic light at an intersection. To ensure safety and follow traffic rules, the ego vehicle should:1. Stop at the red light, if it is currently red, and wait for the green light to proceed.2. If the light is green, the ego vehicle should proceed cautiously, checking for any pedestrians or other vehicles that might be crossing the intersection.3. If the light turns yellow, the ego vehicle should slow down and prepare to stop if necessary.4. If the light turns red, the ego vehicle should stop completely and wait for the light to turn green again before proceeding.5. If the ego vehicle is turning left or right at the intersection, it should yield to any oncoming traffic and wait for a safe gap to make the turn.6. If the ego vehicle is turning left or right at the intersection, it should yield to any oncoming traffic and wait for a safe gap to make the turn.7. If the ego vehicle is turning left or right at the intersection, it should yield to any oncoming traffic and wait for a safe gap to make the turn.8. If the ego vehicle is turning left or right"},{"id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_6","question": "<image>\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is not moving. B. The ego vehicle is steering to the left. The ego vehicle is driving very fast. C. The ego vehicle is steering to the right. The ego vehicle is driving slowly. D. The ego vehicle is steering to the right. The ego vehicle is driving fast.","answer": "The ego vehicle is steering to the right, and it is driving fast."},{"id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_0","question": "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.","answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a car, a truck, a bus, a traffic light, and a building. The car and truck are both on the road, and the bus is also visible on the street. The traffic light is located at the intersection, and the building is situated near the intersection as well. These objects will be important for the driver to consider when making decisions about their driving path, speed, and potential hazards."},{"id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_1","question": "<image>\nWhat is the moving status of object <c1,CAM_FRONT_RIGHT,850.0,484.2>? Please select the correct answer from the following options: A. Reverse parking. B. Back up. C. Going ahead. D. Turn left.","answer": "The moving status of object <c1,CAM_FRONT_RIGHT,850.0,484.2> is \"Going ahead.\""},{"id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_2","question": "<image>\nIs <c1,CAM_FRONT_RIGHT,850.0,484.2> a traffic sign or a road barrier?","answer": "The traffic sign is located on the side of the road, while the road barrier is located in the middle of the road."},{"id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_3","question": "<image>\nWhat actions could the ego vehicle take based on <c1,CAM_FRONT_RIGHT,850.0,484.2>? Why take this action and what's the probability?","answer": "Based on the given coordinates, the ego vehicle is located at the intersection of two streets, with a traffic light on the left side of the road. The vehicle is facing the traffic light, which is green. The ego vehicle could take several actions based on this situation. It could:1. Continue straight through the intersection, following the green traffic light, as it is currently allowed to do so.2. Turn left at the intersection, following the green traffic light, as it is currently allowed to do so.3. Turn right at the intersection, following the green traffic light, as it is currently allowed to do so.4. Stop at the intersection, as it is currently allowed to do so.The probability of the ego vehicle taking any of these actions depends on the specific traffic rules and regulations in the area. In general, the probability of the ego vehicle turning left or right at the intersection is higher than continuing straight through, as it is more likely that the vehicle is turning at an intersection rather than continuing straight. However, the exact probability of each action depends on the specific traffic rules and regulations in the area."},{"id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_4","question": "<image>\nWhat actions taken by the ego vehicle can lead to a collision with <c1,CAM_FRONT_RIGHT,850.0,484.2>?","answer": "The ego vehicle can lead to a collision with <c1,CAM_FRONT_RIGHT,850.0,484.2> by not following the traffic rules, such as speeding, not yielding to pedestrians or other vehicles, or failing to maintain a safe distance from the vehicle in front. Additionally, if the ego vehicle is not paying attention to the road or is distracted, it can also contribute to a collision."},{"id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_5","question": "<image>\nIn this scenario, what are safe actions to take for the ego vehicle?","answer": "In the image, the ego vehicle is driving down a street with a traffic light at the intersection. To ensure safety, the ego vehicle should follow the traffic light's instructions and obey the traffic rules. This includes stopping at red lights, yielding to pedestrians and other vehicles, and maintaining a safe distance from other vehicles. Additionally, the driver should be aware of their surroundings, including pedestrians, cyclists, and other vehicles, and be prepared to react to any unexpected situations. By following these safety measures, the ego vehicle can help prevent accidents and maintain a smooth flow of traffic in the city."},{"id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_6","question": "<image>\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is not moving. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. C. The ego vehicle is steering to the left. The ego vehicle is driving very fast. D. The ego vehicle is steering to the right. The ego vehicle is driving very fast.","answer": "The ego vehicle is steering to the left, and it is driving slowly."},{"id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_0","question": "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.","answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a car, a truck, a bus, and a traffic light. The car and truck are both on the road, and the bus is parked on the side of the road. The traffic light is located at the intersection, indicating that the vehicles need to follow the traffic rules and regulations. The presence of these objects and their positions in the scene will be taken into account by the driver when making future decisions, such as when to change lanes, when to stop, or when to proceed through the intersection."},{"id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_1","question": "<image>\nWhat is the moving status of object <c1,CAM_FRONT,1458.3,492.5>? Please select the correct answer from the following options: A. Going ahead. B. Turn left. C. Stopped. D. Back up.","answer": "The moving status of object <c1,CAM_FRONT,1458.3,492.5> is \"Stopped.\""},{"id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_2","question": "<image>\nIs <c1,CAM_FRONT,1458.3,492.5> a traffic sign or a road barrier?","answer": "The traffic sign is located on the side of the road, and it is not a road barrier."},{"id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_3","question": "<image>\nWhat actions could the ego vehicle take based on <c1,CAM_FRONT,1458.3,492.5>? Why take this action and what's the probability?","answer": "Based on the given coordinates, the ego vehicle could take the following actions:1. Continue driving straight ahead: The vehicle is currently driving on a road, and the coordinates indicate that it is at the intersection of two roads. The vehicle could continue driving straight ahead, following the road it is currently on.2. Turn left: The vehicle could also turn left at the intersection, following the road that is perpendicular to the one it is currently on. This would require the vehicle to make a left turn at the intersection.3. Turn right: The vehicle could also turn right at the intersection, following the road that is parallel to the one it is currently on. This would require the vehicle to make a right turn at the intersection.The probability of the vehicle taking any of these actions depends on the specific situation and the driver's preferences. For example, if the vehicle is approaching a busy intersection or a blind spot, it might be more likely to turn left or right to avoid potential collisions or to maintain a safe distance from other vehicles."},{"id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_4","question": "<image>\nWhat actions taken by the ego vehicle can lead to a collision with <c1,CAM_FRONT,1458.3,492.5>?","answer": "The ego vehicle can lead to a collision with <c1,CAM_FRONT,1458.3,492.5> by not following the traffic rules, such as speeding, not stopping at a red light, or not yielding to pedestrians or other vehicles. Additionally, if the ego vehicle is not paying attention to its surroundings or is distracted, it could also contribute to a collision."},{"id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_5","question": "<image>\nIn this scenario, what are safe actions to take for the ego vehicle?","answer": "In the image, the ego vehicle is driving down a street with a green light. It is important for the ego vehicle to follow the traffic rules and regulations, such as stopping at the red light when it turns red. Additionally, the ego vehicle should be cautious of other vehicles and pedestrians around it, especially when approaching intersections or making turns. It is essential to maintain a safe distance from other vehicles and pedestrians, and be prepared to stop or slow down if necessary. The ego vehicle should also be aware of any potential hazards or obstacles on the road, such as potholes, debris, or pedestrians crossing the street, and take appropriate actions to avoid accidents or collisions."},{"id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_6","question": "<image>\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is driving slowly. B. The ego vehicle is going straight. The ego vehicle is not moving. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving very fast. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving with normal speed.","answer": "The ego vehicle is going straight and driving with normal speed."},{"id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_0","question": "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.","answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a car, a truck, a bus, a traffic light, and a building. The car and truck are both on the road, and the bus is parked on the side of the road. The traffic light is located at the intersection, and the building is situated in the background. These objects will be important for the driver to consider when navigating the road, ensuring safety, and making decisions about the traffic flow."},{"id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_1","question": "<image>\nWhat is the moving status of object <c1,CAM_FRONT,1321.7,495.0>? Please select the correct answer from the following options: A. Turn left. B. Turn right. C. Back up. D. Going ahead.","answer": "The moving status of object <c1,CAM_FRONT,1321.7,495.0> is \"Going ahead.\""},{"id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_2","question": "<image>\nIs <c1,CAM_FRONT,1321.7,495.0> a traffic sign or a road barrier?","answer": "The image description does not provide enough information to definitively determine whether the object is a traffic sign or a road barrier. However, it is possible that it could be either one, depending on the context and the specific details of the object."},{"id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_3","question": "<image>\nWhat actions could the ego vehicle take based on <c1,CAM_FRONT,1321.7,495.0>? Why take this action and what's the probability?","answer": "Based on the given coordinates, the ego vehicle is located at the intersection of two streets, with a traffic light in front of it. The ego vehicle could take several actions, such as:1. Continue straight: The ego vehicle could continue driving straight through the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and following the traffic light's instructions.2. Turn left: The ego vehicle could turn left at the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and following the traffic light's instructions.3. Turn right: The ego vehicle could turn right at the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and following the traffic light's instructions.4. Stop: The ego vehicle could stop at the intersection, waiting for the traffic light to change or for other vehicles to pass. This action has a probability of 0% as the vehicle is already at the intersection and following the traffic light's instructions.5. Turn"},{"id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_4","question": "<image>\nWhat actions taken by the ego vehicle can lead to a collision with <c2,CAM_FRONT,629.2,475.0>?","answer": "The ego vehicle can lead to a collision with <c2,CAM_FRONT,629.2,475.0> by:1. Failing to yield to the pedestrian crossing the street: If the ego vehicle is not paying attention to the pedestrian crossing the street, it may not notice the pedestrian in time and could collide with them.2. Failing to stop at the stop sign: If the ego vehicle fails to stop at the stop sign, it may continue driving without checking for oncoming traffic or pedestrians, increasing the risk of a collision.3. Failing to yield to the traffic light: If the ego vehicle fails to yield to the traffic light, it may continue driving without checking for oncoming traffic or pedestrians, increasing the risk of a collision.4. Failing to maintain a safe distance from the vehicle in front: If the ego vehicle is too close to the vehicle in front, it may not have enough time to react to sudden changes in the vehicle's speed or direction, increasing the risk of a collision.5. Driving too fast for the road conditions: If the ego vehicle is driving too"},{"id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_5","question": "<image>\nIn this scenario, what are safe actions to take for the ego vehicle?","answer": "In the image, the ego vehicle is driving down a street with a sidewalk on the side. To ensure safety, the driver should:1. Maintain a safe distance from the sidewalk, especially if pedestrians are present.2. Be cautious of any pedestrians or cyclists who may be crossing the street or using the sidewalk.3. Obey traffic signals and signs, as well as any posted speed limits.4. Be aware of any potential hazards or obstacles on the road, such as potholes, debris, or other vehicles.5. Be prepared to stop or slow down if necessary to avoid collisions or accidents.By following these safety measures, the driver can help ensure the safety of themselves, their passengers, and other road users."},{"id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_6","question": "<image>\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving fast. B. The ego vehicle is steering to the left. The ego vehicle is driving very fast. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving with normal speed.","answer": "The ego vehicle is slightly steering to the left, and it is driving with normal speed."}
]

五、训练（finetune）demo

训练需要修改finetune_data_config.yaml中关于训练数据的路径

demo可以直接利用test_llama.json进行训练。如果要复现baseline需要下载nuScenes并且转化出train_llama.json，后续会提及，这里先跑通代码。

同理，ckpts是adapter的，output path是work_dirs的路径，随便填，会自动创建，命令执行后会在ouput_path下生成logs并将终端的信息打印进去（感觉是商汤的老传统了，跑脚本都不喜欢在终端看，喜欢重定向到log文件去看）。同推理，单卡需要修改--nproc_per_node=1

# /path/to/llama_model_weights, /path/to/pre-trained/checkpoint.pth and /output/path need to be modified by your path
# make sure you are under ./challenge/llama_adapter_v2_multimodal7b
.exps/finetune.sh \
/path/to/llama_model_weights /path/to/pre-trained/checkpoint.pth \
finetune_data_config.yaml /output/path

我这边在终端打印出了log

默认4epoch 训练后会在output_path下保存 e1和e4的模型。

这时候可以重新推理。将原本是llama-adapter-v2的ckpt改成finetune保存后的checkpoint-3.pth。

然后发现结果好很多，主要是对于一些选择题，能正确选出答案，原本zero-shot的llama只是倾向于生成答案，并不会做选择题。说明train=val的这个训练没问题。

六、训练（finetune）

接下来可以用全量数据进行训练，复现baseline。

首先按照上面的数据表格下载nuScenes-DriveLM。笔者这里用的v1_1版本。后续可能还会更新。

下载完drivelm_nus_imgs_train.zip后进行解压unzip，得到一个nuscenes文件夹，mv到challenge/llama_adapter_v2_multimodal7b/data/目录下，得到

然后回到./challenge/目录下

1.修改extract_data.py，将注释掉的那两行root_path改为上述表格里DriveLM完整的annotation数据:train。save_path是最后保存的json文件名，这样就生成了图-QA对的训练数据。

python extract_data.py

得到类似repo自带的challenge/test.json形式的json文件。

2.为了对齐eval的数据格式，需要修改convert_data.py

python convert_data.py

为什么要进行这一步？看了代码发现是因为eval的时候对于一些moving status采用选择题的形式给出，但是train.json原始的答案是直接给出状态描述，如果不进行这一步，模型就不会做选择题了。

吐槽下，这里的命名很有迷惑性。。。_eval.json不是用来eval的，而是代表形式跟eval对齐（否则训练集和验证集格式/分布不统一，后果懂的都懂）。其实这一步本来可以归到extract_data.py里或者在v1_1_train_nus.json里修改格式统一，没必要专门来一个步骤单独处理一种QA转化为选择题。

3.接着为了进行LLaMA的finetune，需要转化为LLaMA所需的格式。

还是在./challenge/目录下，修改convert_data.py并执行。

python convert2llama.py

最后一步，将finetune_data_config.yaml里的路径改为train_llama.json。然后执行上面的finetune脚本

# /path/to/llama_model_weights, /path/to/pre-trained/checkpoint.pth and /output/path need to be modified by your path
# make sure you are under ./challenge/llama_adapter_v2_multimodal7b
.exps/finetune.sh \
/path/to/llama_model_weights /path/to/pre-trained/checkpoint.pth \
finetune_data_config.yaml /output/path

用默认的8卡A100（显存占用34G）跑4epoch bs4。每个epoch大概十分钟，总共40分钟。