ML-DevOps Integration
This is my first small project of Machine Learning with DevOps. In this project I have prepared a Python code for CNN. It is written in such a way that every time when it is executed it will change its architecture by choosing the different parameters and then train the model to achieve the accuracy at least more than 80%.
The complete workflow of the project is like this.
1. Create container image that’s has Python3 and Keras or numpy installed using dockerfile
2. When we launch this image, it should automatically start training the model in the container.
3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins
4. Job1 : Pull the Github repo automatically when some developers push repo to Github.
5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the CNN processing).
6. Job3 : Train your model and predict accuracy or metrics.
7. Job4 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture and also retrain the model or notify that the best model is being created
8. Create One extra job job5 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left
Step 1:-
Create container image that’s has Python3 and Keras or numpy installed using Dockerfile
Build the Container Image
docker build -t model:v1 .
Job1:-
This job will pull the Github repo automatically when the developer will push the code to the repo and this is done by using Poll SCM.
Job 2:-
This job will check the program file and by looking at the program file it will automatically start the respective machine learning software installed interpreter install image container to deploy code.
Job 3:-
This job will start training the model and will also print the accuracy.
Job 4:-
This job will check the accuracy and if it is less than 80% , then tweak the machine learning model architecture and again retrain the model or notify that the best model is being created but in my case the accuracy is more than 80% so it will notify that the best model has been created. For notification I have written a python program which will be executed by this job and that program will send an email to the developer.
Step 3:-
I have created a job chain using the build pipeline plugin in jenkins that will run all these job.
Now when the developer will push repo to Github then the job1 will be automatically triggered.
The output of job chain is shown below.
Step 4:-
In this step one extra job is created for monitoring. If the container where the app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left.
Github link:-
