Easily create a cloud batch job using AWS and Boxfuse
I was looking for a way as simple as AWS Lambda to run a batch job on AWS.
These were my requirements:
- fully automated compilation, deployment and scheduling
- Not with Docker, it’s too much hassle for what I want to achieve
- on AWS, because they rock
- Simple, plain vanilla Java, no boiler plating, no imposed libraries nor frameworks, I’m getting too old for that, certainly, when all I need is a main function…
Enters Boxfuse: a super easy way to develop, test and then push web apps, groovy, go, nodejs and many other application flavors to the cloud. And fortunately you can as easily push Executable JARs to the cloud.
So here’s what I did:
- I installed VirtualBox for local testing on my iMac
- created an account and defined an application of worker type @ Boxfuse
- created an IntelliJ maven based project containing not much more than a main class that runs indefinitely
- configured Maven in my project to support boxfuse
- ran it locally in VirtualBox (and ok, I admit it, fiddling a little to get my program working properly)
- pushed it to AWS
- padded myself on the back and started drinking a local Knots beer [sce emoji=”beer”/] 🙂
If you want more details on the steps above, here’s a little more…
Create the Account and define an application
Create an account here, learn about application types here.
I went for the worker application type. On the smallest AWS instance available (at the time of this writing that was the nano.
Configure Maven
I don’t like typing the same things more than 3 times, so luckily Boxfuse comes with a maven plugin. I followed the instructions here to automate running the maven application on my iMac.
[code type=xml]
Running the application in VirtualBox
Now, using the following command I can clean, package and run the project locally on my VirtualBox:
mvn clean package boxfuse:run
An here’s the result:
[INFO] ------------------------------------------------------------------------ [INFO] Building CreepyCrawler 1.0 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- boxfuse-maven-plugin:1.22.2.1149:run (default-cli) @ CreepyCrawler --- [INFO] Boxfuse client v.1.22.2.1149 [INFO] Copyright 2016 Boxfuse GmbH. All rights reserved. [INFO] [INFO] Account: patrizz (Patrice Kerremans) [INFO] [INFO] Fusing Image for CreepyCrawler-1.0.jar (Jar) ... [INFO] Image fused in 00:13.443s (87481 K) -> patrizz/quitus-crawler:1.0 [INFO] Launching Instance of patrizz/quitus-crawler:1.0 on VirtualBox ... [INFO] Forwarding http port localhost:10080 -> vb-2ff576a8:80 [INFO] Instance launched in 00:02.677s -> vb-2ff576a8 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 33.240 s [INFO] Finished at: 2016-11-06T11:06:16+01:00 [INFO] Final Memory: 47M/1237M [INFO] ------------------------------------------------------------------------ Patrices-iMac:CreepyCrawler patricekerremans$
Just a wiff longer than half a minute and my Executable JAR containing a Main class was running inside a container.
Deploying to AWS
That’s actually the easiest part, just specifying the “prod” environment was enough to have it run on AWS:
Patrices-iMac:CreepyCrawler patricekerremans$ boxfuse -healthcheck=false -logs.tail=true -logs=true -env=prod run quitus-crawler Boxfuse client v.1.22.2.1149 Copyright 2016 Boxfuse GmbH. All rights reserved. Account: patrizz (Patrice Kerremans) Pushing patrizz/quitus-crawler:1.0 ... [#######################################-] 86844/88380 K
Unfortunately I hadn’t properly set up the Boxfuse AIM role, so I had to go back to the AWS console and fix it. The second time around everything went as expected, though:
Patrices-iMac:CreepyCrawler patricekerremans$ boxfuse -healthcheck=false -logs.tail=true -logs=true -env=prod run quitus-crawler Boxfuse client v.1.22.2.1149 Copyright 2016 Boxfuse GmbH. All rights reserved. Account: patrizz (Patrice Kerremans) Creating Log Group boxfuse/prod ... Creating Log Stream boxfuse/prod > patrizz/quitus-crawler ... Creating IAM Role boxfuse-cloudwatchlogs-agent-role ... Creating security group boxsg-patrizz-prod-quitus-crawler-1.0 ... Creating Launch Configuration boxlc-patrizz-prod-quitus-crawler-1.0 ... Creating Auto Scaling Group boxasg-patrizz-prod-quitus-crawler-1.0 ... Waiting for Auto Scaling Group boxasg-patrizz-prod-quitus-crawler-1.0 to launch 1 t2.micro Instance ... Auto Scaling Group: i-cd960e70 [Pending] Auto Scaling Group: i-cd960e70 [InService] Deployment completed successfully. patrizz/quitus-crawler:1.0 is up and running. Patrices-iMac:CreepyCrawler patricekerremans$
And here’s the application running in the AWS console:
After correcting a final user related issue (the Boxfuse temporary user didn’t have access to a AWS service that’s being used in the Main class) the logs stopped showinf errors 🙂
Conclusion
Using Boxfuse made it incredibly easy for me to deploy my batch job locally in VirtualBox for easy testing with a minimum of dependencies and worries. Sending the same off to the cloud was then a matter of seconds.
Impressive! [sce emoji=”yes”/]