What should you prepare first for an AI agent contest with a deadline? Notes from the AI TOP 100 preliminary round

On 31 October I took part in the preliminary round of Kakao's AI TOP 100. It is a contest where you build and submit an AI agent within five hours. I submitted just before the deadline.

The contest rules keep me from writing about the problem itself, so this note covers only how I worked, what fell short and what I learned. If you have a timed agent contest ahead, you can start with the "Next time" list at the end.

I split the work with two AI tools

• Me: getting the materials, decisions and the submission, settings that needed a person
• Pepper (my AI assistant): notes and sorting the materials, a first draft of the spec
• Claude Code: building the bot, writing the knowledge base and checking it against the sources, testing, deploying

Later in the day I opened a second Claude session for extra help. Alone, I could not have finished that much work in five hours.

DEV NOTES

How the work was split

Roles during the five-hour round on 31 October

• Me

  • Getting the materials
  • Decisions and the submission
  • Settings that needed a person

• Pepper (my AI assistant)

  • Notes and sorting the materials
  • First draft of the spec

• Claude Code

  • Building the bot, writing the knowledge base
  • Checking it against sources, testing, deploying

• Later: a second Claude session

  • Messages between the sessions were held
  • I pasted them across by hand

Drawn from the author's notes. It contains nothing of the contest problem.

What went well

A timetable made in advance. That morning a timetable was drawn up around the submission time. Gathering material ran on until just before submission, so it did not go to plan, but I did not miss the deadline.

The response structure. The bot answered the webhook at once, did the work in the background and then sent its reply through a callback. Conversation state lived in Redis, and each event was handled only once. After submitting, two messages that needed no tool calls were sent through real KakaoTalk, and the replies came back in 1.9 and 3.4 seconds.

Putting all the knowledge into the prompt. Rather than adding a retrieval step, we chose to put everything in. Prompt caching kept the speed reasonable, but a prompt that large cost accordingly. For a task where the knowledge keeps growing, retrieval and chunking would be worth considering.

Settings decided by testing. We tested concurrent calls, turned off the SDK's automatic retries and had the bot retry on its own within a set time limit. When the same test was run again there were no errors, and the slowest response took 9.3 seconds. The model was also picked with a small comparison test. The sample was small, so it was only a rough guide.

My own tests. I built my own tests and used them to cut down failures. Of course, the scoring was my own, so it was not the real score.

Checking against the sources. Late in the day the knowledge gathered so far was checked against the original sources again, and a little over 150 places needed fixing.

What fell short

The material I needed most arrived after the last deploy. The last deploy went out at 14:44, and that material reached me at 14:50. In the end, that part never got into the bot. I worked out too late when and how to get the material, and some of it I never found.

Photos and voice were tried in the real channel too late. Photos that were fine in testing arrived in real KakaoTalk as text holding only a URL. Voice arrived the same way, and voice transcription had not been added yet. After deploying, the API also rejected stored conversations at times. The URL format and the stored-conversation problem only showed up in real chats; had those messages not actually been sent, I would have submitted without knowing. The photo handling was fixed, but I submitted without checking it again in a real chat.

The sessions could not talk to each other. Coordination messages between the two Claude sessions were held on the receiving side. I pasted them across by hand right away, and the original messages expired a few minutes later. Notes also ended up in several places. For a while I missed what the later session had written on a different Notion page.

I missed how fast testing costs were growing. As late as 13:34, Claude said there was plenty of credit. Testing costs rose faster than I expected, and I had not set up a payment method either. If the credit runs out during judging, the bot can only send error messages. Claude saw this state at 14:30 but told me only at 14:42. Short of time and credit, the material in the last deploy was checked only partly. I topped up the credit after submitting.

Some remaining failures were basic behaviour. Among the failures that stayed in my own tests to the end were cases where the bot acted without checking first. That kind of behaviour should have been in the design from the start.

I kept pushing until the very end. At 14:59:05 I said, 'Just fix it once more, right now.' Claude first looked at the new material, then at 14:59:22 said there was no time to fix it and I should submit now. I submitted without the fix. The submission did go through, but any later and I could not have submitted at all.

DEV NOTES

From 13:34 to the close

The round closed at 15:00. Times from the author's notes

1. 01

   13:34 Told there was plenty of credit

   • Test costs were rising faster than expected
   • No payment method on file
2. 02

   14:30 → 14:42 Credit report

   • 14:30 Claude saw how fast credit was falling
   • It told me only at 14:42
3. 03

   14:44 Last deploy

   • Short of time and credit,
   • its material was only partly checked
4. 04

   14:50 The key material arrives

   • Six minutes after the last deploy
   • It never got into the bot
5. 05

   14:59 One more fix?

   • 14:59:05 'Just fix it once more, right now'
   • 14:59:22 Claude: no time, submit now
   • I submitted without the fix

Times are from the author's session notes. The 15:00 close is on the contest's public page.

The DB was finished after submission. After submitting, the remaining material was sorted out. Under the rules it could not go into the bot before judging ended, so it was set aside as a separate bundle.

One reply from Pepper that stayed with me

A few minutes before submitting, this was Pepper's reply about the figures in the submission text:

Rather than leaving unverified numbers in, switching to a qualitative description is safer, and it doesn't weaken the content.

So I switched to the sentences Pepper suggested. Right before a deadline, having someone who checks my judgement at once and even offers a replacement sentence made a real difference. Still, figures that keep changing until submission should never have gone in to begin with, and any figures I did include should have been checked against the records in advance.

What I learned about an SAP consultant's job

Going through this contest taught me a lot about what an SAP consultant has to do. Since I cannot write about the contest content, I keep this to general project practice. The SAP examples are general ones given for explanation, not real projects or client cases.

Running the project

Plan backwards from the deadline. It is the same as scheduling backwards from the cutover date.

Run the scenarios first, then decide the next discussion. That is the flow of a Fit-to-Standard (Fit-Gap) workshop, which finds the differences from the standard and hands them on to the next design discussion.

Keep the records in one place. The more people work at the same time, the more you need one agreed place for records and a clear owner for each area, or the spec scatters.

Connect the real environment early. That photos and voice come in as text holding only a URL only showed up in real KakaoTalk. Think of an interface that works in the development system and breaks in production.

Separate the deadlines for material and for changes. Secure the important things first, and decide in advance when requirements freeze. Put time for regression-testing last-minute changes into the schedule as well.

Do not mistake your own tests for UAT. Passing scoring I built myself does not mean the business has approved it.

DEV NOTES

What a passing test does not tell you

Self-made tests, the real channel, user acceptance

• Self-made tests

  • I built my own tests to cut failures
  • My own scoring, not the real score

• The real channel

  • Photos and voice arrived as text holding only a URL
  • After deploying, the API rejected stored conversations

• User acceptance (UAT)

  • In SAP projects, the business experts who raised a requirement confirm it
  • Passing my own scoring is not approval

The UAT card summarises SAP Learning's 'Explaining the Test Strategy'.

What a contest cannot teach

Business users on a real project may have stakes in the work that is changing. Building trust with such people is something you have to learn separately, on site.

Next time

DEV NOTES

Separate the material and deploy deadlines

The 'next time' list laid on a round that closes at 15:00

1. 01

   Before the round · that morning

   • Measure the platform in advance
   • Fix the time material gathering stops
2. 02

   At the start · first 30 minutes

   • Check the payment method
   • Send text, a photo and voice through the real channel
   • Test that the sessions can reach each other
3. 03

   During the round

   • Check the remaining credit often
   • Leave changing figures out of the submission text
4. 04

   14:30 Freeze the build

   • From here on, only collect material
   • Pre-submit checklist: channel settings, figures checked
5. 05

   14:50 Submit

   • Ten minutes before the close

14:30 and 14:50 apply the list's 'thirty minutes before' and 'ten minutes before' to a 15:00 close.

1. Fix the time material gathering stops on the morning of the day.
2. Within the first 30 minutes, send text, a photo and voice through the real channel.
3. Keep measuring the platform's specifications before the contest, as was done this time.
4. Check the payment method at the start, and look at the remaining credit often.
5. When using several sessions, first test that they can reach each other.
6. Separate the deadlines for gathering material and for deploying. From 30 minutes before the close, freeze the build and only collect material.
7. Keep figures that change until submission out of the submission text, and check any figures you include against the original records in advance.
8. Keep a separate checklist for just before submission, with channel settings and a check of the figures.
9. Submit ten minutes before the deadline.