Build an Android app to recognize flowers

9 Likes comments off

[MUSIC PLAYING]

 

KHANH LEVIET: Hi, everyone.

I’m Khanh from the
TensorFlow team.

And today I’d like to show you
how to train a TensorFlow Lite

model that can recognize custom
images using your own data set.

And then my colleague
Hoi will show you

how to integrate the model into
an Android app using the new ML

Model Binding Plugin
in Android Studio.

In this example,
we’ll build an app

that can recognize five
different types of flowers,

daisy, rose,
sunflower, and so on.

Now, let’s get started.

We’ll start with training
our model in Google Colab.

Google Colab is an
environment that

allows you to use CPE for free
to quickly train your machine

learning model.

By using Colab, you don’t
need to install any software

to your computer.

You only need to use
your web browsers.

You can go to this Colab
Notebook from this link.

This is also included in
the video description.

Let’s start by installing
TensorFlow Lite Model

Maker to Colab.

Model Maker is a tool
that allows easy training

tests for Lite models
with no machine learning

expertise required.

Now the installation
has finished.

Let’s import the
required Python package.

 

Let’s download our training
images from TensorFlow

to our Colab machine.

 

You can also download
the training data

from this URL to your computer
to explore the training data

set.

You will find that
it contains images

of five different
types of flowers

organized into each folder.

So for daisy, we have several
hundreds of images, dandelion,

rose, and so on.

And if you have training
images that you want to use,

you can also upload
them to Colab.

Just go to the Files
tab right here,

click on the Upload
button, and then you

can upload your images.

 

Next, we upload our
images into Model Maker.

Here you can see that
I split the data set

into train data and test data.

The number 0.9 here means
that we’ll use 90% our data

for training the model, and
the remaining 10% of the data

to test its accuracy.

I’ll explain more about
that later in this video.

Now the loading
process has completed.

Here we have 3,670 images
in five categories, daisy,

dandelion, roses,
sunflowers, and tulips.

Now we have our
training data set.

Let’s train our TensorFlow Lite
model with the flower data set.

To train faster,
make sure that you

have enabled GPU in your Colab.

Go to Runtime,
Change Runtime Type,

and make sure that GPU
is already selected.

 

You can see that here
I use only the train

data, which is the 90%
portion of our data set

that I loaded earlier.

 

You can see that the model
is now being trained,

and the accuracy is improving.

 

Now, after about 90
seconds of training,

I have a model with 93%
accuracy, which is quite good.

You will probably see your
accuracy number a little bit

different from my
accuracy number,

because before training,
the model is initialized

with random weight values.

So your initial
value is very likely

to be different to my
initial value, which

led to a slight difference
in the final accuracy.

And next, let’s
evaluate our model.

 

You can see here that I
used the test data, which

is the remaining 10% our data
set that I left out earlier.

The reason why I did that
was to keep some data

that the model didn’t see during
training so that we can test

if the model can
generalize well to new data

that it has never seen before.

Our test accuracy is about
92%, which is a little bit

lower than our
training accuracy,

but still pretty good.

And finally, let’s export
our TensorFlow Lite model

to integrate it into
our Android app.

 

You can see that the export
process has finished,

and now I have the TensorFlow
Lite model in my Colab.

Let’s download the model
to our local computer.

 

And then I’ll share
the model with Hoi,

my Android local friend.

HOI LAM: Thanks, Khanh.

I’m Hoi from the Android team.

Today, I’ll walk you
through the steps required

to integrate the TF Lite
model that Khanh has just

built into an Android app.

The first step is to set up.

Go to developer.androi
d.com/studio/preview and select

Download 4.1 Beta
1 as your option.

You have the option to
keep several versions

of Android Studio
installed at the same time.

If that’s what you want to do,
please scroll down on this page

and refer to the documentation
saying Run Alongside Stable

and click on Install
Both to find out more.

Once you have Android
Studio Beta installed,

go to the URL link in the
video description down below

  Android Studio: Using and Creating Them

to download his Colabs
Android Studio project.

It contains the basic
CameraX functionalities which

we will use to feed the model.

Just like any good
cooking show, I’ve

already downloaded the
Android Studio Beta,

installed it, and also
downloaded the Android Studio

project that we’re
going to work on.

So let’s take a look.

So this is Android Studio
version 4.1 beta 1.

Let’s open up the project
and see what we have.

 

So let me just find
it, click Open.

And here, Android Studio is
opening the project itself.

It’s indexing all
the files, and this

might take a while
depending on what you

have set up on your machine.

So it looks like
everything is fine.

OK.

As we can see, we
have two modules here

in this project,
Start and Finish.

So under Start is the projects
that we’re going to work on,

and Finish is basically
the finished project.

So if you ever get stuck in this
particular Colab, go to Finish

and you’ll be able to see
what the code should be.

So we’ll be working with
Start throughout this project.

And one of the
things that you can

do at the very beginning
of this project

is to just run
the Start project.

So just click here to run it.

So Gradle is building
and it’s now installing.

And this is what you should
see on your screen, which

is a preview of what
the camera sees.

And at the bottom,
you have three labels

that are printed right next
to some random numbers that

are fake labels.

And that’s the
start of our Colab.

OK, let’s stop that and go back.

And one of the ways to
really easily navigate

through this Colab is by
viewing all the to-dos.

So if you click on View,
Tools window, To-Do,

you’ll see a list
of different items

within the projects of
what we needed to do.

And one of the
ways to easily see

which ones are under the Start
project is go here, Group By,

and select By Module.

And here, order to-do items
and group by Finish and Start.

Under Start, these
are the actual places

where we expect to insert code.

The first to-do is actually
nothing to do with code,

but about importing
an actual model.

So let’s do that.

Let’s import the model
that Khanh have created.

OK, so the easiest thing to do
to import the model that Khanh

had just produced
are the new tooling

that you get with
Android Studio 4.1.

To start, right click
on a module where

you want to put the
particular TF Lite model,

select New, select
Other, and here, we

have the TensorFlow Lite model.

Click on that and then
select the model location.

Open it up.

I’ve got it under
here, under Download.

So one thing you have
noticed is that I’ve actually

changed the name of the
model that I’ve downloaded.

So Khanh named it model.tflite.

If you import it without
changing the name,

then the classes that generated
for you to use that model

will be model.

And model is an
awfully common name,

so in order to reduce
confusion, always

change the name of your
model into something

that is a little
bit more meaningful,

and you will see
that further down

in this particular screencast.

So now select that
model, click Open.

Here we have the path.

And what else have we got here?

So the other things that
this import module do

is not just copying
the TF Lite model,

but actually insert
the necessary settings

to use the model.

So there are two things here.

One is to enable a
new build feature

that is in Android Studio
4.1, the ML model binding.

In addition, we have
some Gradle import

and dependencies that
we will need in order

to use the ML model binding.

So they’re here, and they
will be inserted automatically

into your [? build-out ?]
Gradle file.

So no more messing around
with the setting file.

Another thing that
it has is if you

want to use the GPU acceleration
through the TensorFlow Lite’s

GPU data kit.

You’ll need to do this.

We’ll do that later on in
this particular screen cast.

So don’t worry about
it for the time being.

So let’s click Finish.

So after you have
imported the model,

this is the first thing
that will greet you

in the new Android Studio 4.1.

You can see from a high
level, you know, what it does.

So for example, the input
is an image of float.

And then, the
output is 5 labels.

  Android Studio 3.0: Android Profiler

And in this case, those
are the five labels

of flowers that
Khanh have trained.

And if you go
slightly down here,

you will also see
the sample code

of how to actually use the
new ML binding functionality.

So you initialize an
instance, create an image

from either the
camera or a bitmap.

And then, you
process the output.

And you get the
probability as a list.

So here, we have an
image analyzer class,

where we are using the
CameraX functionality.

And for each frame
that a CameraX gets,

it will basically fit it
into the analyze method here.

So before we can use that, we
will need to initiate the model

itself.

So let’s do that.

Let’s create a private
instance of the flower model

that we have just import
and call it flower model.

And the way to initialize
it is like this.

So just call it
FlowerModel.NewInstance,

and then feeding in the context.

 

Here we go.

So you’ll notice that here we’re
basically using the same name

as the TF Lite model itself.

So name your TF Lite model
carefully before you import,

so that you will have a
much better time in terms

of finding out which
model you should be using

and basically give
it an intuitive name.

So now, that we got an
instance of the model,

what do we do with
each of the frame?

The first thing to do
is to convert the image

into a format that
we can process,

namely the TensorFlow Lite
or the Tensor image format.

So let’s create a
variable, call it TF image

and create an instance of
Tensor image from bitmap.

So if you’ve done this before
with TensorFlow interpreter,

you have noticed
that previously, you

might need to convert your
image into byte array directly.

Here, you don’t actually
need to do any of that.

The input can be bitmap.

And in this case,
we have image proxy.

So in this case, we need to
convert that into a bitmap

before feeding it in.

So I’ve created a helper method
here to say 2 bitmap, feeding

in an image proxy.

And this is all you need to do.

If instead of using
a camera as an input,

and you want to load up a image
that’s already on the phone,

you can feed that
bitmap directly here.

 

Next, we’re going to feed
this particular image

into the model for processing.

So we will create a
output object, so outputs.

And we’ll utilize
the flower model

that we have initialized
before and process it.

 

Process the TF image.

 

And what we can get
after we process an image

is basically a list of
category objects, which

include both the label as
well as the probability

of the output or our detection.

So select that.

And the next, what we
want to do is to sort it,

so that the highest probability
result is at the top.

And then, we want to
limit it to, let’s say,

the top three items.

And how do we do that?

So there is this really
helpful Kotlin feature

that will allow you
to solve for the list

by specifying the attribute
that you want to sort on.

So what we do is, I go, .apply.

 

And here, I’m going to
say, sortDescending.

Because what we want to
do is to sort the list,

so that the highest
probability is at the top,

and the lowest probability
is at the end of the list.

So let’s do that.

And here, we need to specify
which field we want to sort by.

So I would type in, it, dot.

So here, you can
see the two fields.

Of course, we’re not
going to sort by label.

But instead, we’re going to
sort by the score, which is,

in this case, the probability.

So here we go.

We have got the list of
the different categories

together with the probability.

And we have now
with this one step,

sorted it by the
probability itself.

The next thing
I’m going to do is

I’m going to cut the
list down in size,

so that it only display
the top three results.

And the way to do that is to use
another Kotlin feature, .take.

And we can say, the
maximum display results.

So this is a constant that
I’ve defined as being three.

You can absolutely change
that to whatever you want.

In this case, it may
not be helping a lot,

because the list itself
is only five items.

We’re only training a model
with five different types

of flowers.

But you could
imagine that if you

have a model that have a
hundred different categories,

or maybe even thousand, or
maybe even tens of thousands,

  Getting started with Firebase on Android

or hundreds of thousands,
this could really

help you cut down on the
number of results that don’t

get passed on and processed.

So please do take
time to consider this.

So you can cut down the
amount of processing

that you need to do.

So after this, what
I need to do is

to convert those category
results into data objects

that my particular
application will understand.

So I’ve created a data
class called recognition

with a label and probability.

So I just need to convert
the results that I have.

In this case, it’s the three
highest probable flowers

that we have into those
recognition objects.

So let’s do that.

So it’s four output in outputs.

 

And I’m going to go items.

So this is the list
of items that we’re

going to return back to the UI.

So items.add,
recognition, output.label,

and then output.score.

And here we go.

One last thing before
we can run the app

is to take out this
section, which I previously

used to generate basically fake
result, placeholder results

so that you can see
the app running.

So let’s comment that.

Automatic, it’s
command and then slash

key to comment those
particular blocks out.

So here we go.

We have our app.

Let’s see it running.

 

As you can see, basically when
you look at the sunflower,

then the sunflower
probability increases

and is working as we expect.

So one last thing I’m going
to do before I sign off

is to show you how you could
use the GPU accelerator.

So to do that, the first thing
we need to do is to go back

to the [? build-out ?]
Gradle file.

And under here, so
under the to-do,

we can see that, hey, we need to
add the optional GPU data kit.

So in the previous color import
dialog, you can just tick a box

and get it.

But in this case, because
we didn’t tick that box,

we will need to manually
perform this step.

So super easy.

Just type in implementation and
is org.TensorFlow:TensorFlow.

Let’s go -lite-gpu:2.2.0.

And let’s press Sync.

Yay.

OK, it looks like it’s done it.

So this step basically imported
the TensorFlow Lite GPU

into your project.

And the next thing that
we’re going to do is use it.

So go to to-do 6.

Here, we need to create an
options object for the model

itself and say,
hey, we really want

to use the GPU accelerator.

So to do that, create a private
variable and call it options.

And this time, we’re
using this model class.

So under the TensorFlow
support library,

you need to choose this
one, org.TensorFlow.l

ite.support.model.

So that’s the one that we want.

And then under that,
there’s options.

And there’s a builder
for that option.

 

First one you could do
is to say, setDevice.

You can also see that
there is an option, set

the number of threads.

So setDevice is
the option to use

different types of accelerator.

So in this case, we’re
going to use a GPU.

You can also choose
an API if you want.

So setDevice, and then we go
model.device., in this case,

gpu.

And last but not
least, we need a build.

But just before
that, I just want

to highlight this other
option that we have,

set the number of threads.

So if instead you want to
use different CPU threads

to run your model, you can also
select this one and then type

in as an integer how many
threads you want to spin up.

And that’s all you need to do.

In this case, we’re going
to leave it with the GPU

and build this object.

So at the moment this option is
not used and the way to use it

is by just doing comma
and say, options.

So there, you have
it, just two lines,

or one line of code, and
then a method change.

And now, you can use
the GPU to accelerate

the running of your model.

So let’s take a look.

 

Fantastic.

So the model is running again.

But this time round,
it has now got

the GPU acceleration enabled.

So to sum up, make your own
model and running it on Android

has never been easier.

You can use the model
maker that Khanh

has shown you to
train that image

classifier and the
Android Studio tooling

I’ve demonstrated to put
it to work on Android.

Check out the video’s
description for more.

See you next time.

[MUSIC PLAYING]

 

 

You might like

About the Author: admin

You cannot copy content of this page