Using CloudComPy to extend CANUPO for point cloud classification

[lweidner]

Hi,
I think it is time for CANUPO to get an upgrade to reflect the last 10 years of development in point cloud semantic segmentation.

While very powerful for some things, CANUPO:
- is limited to binary classification (or rather, multi class classification is cumbersome)
- uses only geometric features
- uses LDA, where Random Forest and various neural network architectures are by far the most common in research
- confidence outputs are not that useful (I have to guess what a good confidence threshold will be before hand)

I found that being able to add new feature types (color, texture, roughness, verticality), use different algorithms (random forest, neural network), and inspect class probabilities interactively, I was able to get much better classification results and apply classification to more problems.

The problem is, many of my colleagues are familiar with CANUPO and have used it with mixed results, since it is the only readily available GUI classification tool out there and open-source. I think many people could make use of better classification GUI tools without needing to become machine learning or coding experts.

Wondering if anyone has thoughts on this. Is it possible to create a CC plugin using CloudComPy? As someone who is really not a C++ coder and only a basic python user, I don't know if I'm capable of building something on my own.

I have a demo using CloudComPy here:
https://github.com/lmweidner/terpunkto/ ... unktoPy.py
It inputs a training point cloud from CC (any number of classes), then you say what scales you want to calculate, whether to include color features, and then it trains a Random Forest to predict the label of the whole point cloud.

[daniel]

First, thanks for this nice example of using CloudComPy!

And that's funny, because the team which created Canupo is also working for quite some time now on a new classification plugin based on Random forest! And it's much more powerful than Canupo, as it lets you mix various features (scalar fields, etc.) or even use 2 different clouds of the same area with different features, etc.

Of course this plugin has so many possibilities that it might also be harder to 'tame' ;)

I'll ask the team if they have some plans to share this plugin with the rest of the world.

[Dimitri]

Indeed we have developped a plugin (called 3DMASC) with colleagues in applied maths, and Daniel for the (paid for) implementation. This plugin has actually been ready for 2 years...it does all you suggest (using the OpenCV library for the random forest training in CC), is command line ready and even more, but the GUI is not as friendly as CANUPO given the complexity in terms of classes and attributes. Getting something as friendly as CANUPO in terms of GUI, is not as much a problem of interface as a problem of the increased complexity when you can have any number of classes and > 200 attributes to play with. Yet we designed the plugin to be extremely versatile in its range of applications (e.g., using it solely to compute the features quickly using the power of CC for neighbours search, and use python for the learning part). There might be a way to generate a stripped down version with a more user friendly advanced GUI when working on 1 point cloud and a reduced set of attributes.

Now why in the hell did we not release 3DMASC: the problem is that it was part of PhD project started in 2017 that never get finished, and I never had the time to publish a paper on it. There's a conference paper in french (https://hal.archives-ouvertes.fr/hal-02070083/) available that shows the basic principles (multi-scale attributes, multi-point clouds and random forests) applied to topo-bathymetric LiDAR with a prototype python implementation. I'll try my best to get something published this year, or find a way for the plugin to be citable somehow in order to release it (all ideas are welcomed).

Some aspects of the plugin were released : you may actually wonder why, all of the sudden, advances geometric features based on PCA were available in CC about 2 years ago ;-) well, this was a declination of some of the features available in 3DMASC ;-)

We can contemplate providing the plugin to a few interested users in the context of collaboration while the paper is not published. We have a proto-documentation, even though the learning curve is way steeper than CANUPO ;-)

[lweidner]

Hi Daniel and Dimitri,

Wow, I should have posted here sooner! That's big news. And makes sense of why geometric features suddenly became available in CloudCompare :) Actually I was in grad school around the same time, 2018 to 2021, and since I wasn't aware of any classification tools other than CANUPO, I built my own in MATLAB (painfully slow and inefficient!!!) and used it for my research. It does seem though that my conclusions in terms of accuracy and the usefulness of different features would be similar to your findings with 3DMASC.
(a few publications:
https://doi.org/10.1016/j.enggeo.2021.106344
https://doi.org/10.1016/j.isprsjprs.2021.04.001
https://doi.org/10.1016/j.enggeo.2019.105326 <-- this one in particular compares CANUPO with Random Forest
https://doi.org/10.1007/s10346-021-01723-4
)

I understand the challenges related to making a GUI that is comprehensible. If CANUPO is any indication, I'm sure you will get lots of forum post questions if you release 3DMASC :) In my case, I had to make decisions on which parameters to expose to the user and which ones were better off being hard-coded. I could write a lot more thoughts about GUI design and what features I think are most important.

On the other hand, I personally know many people in academic and industry positions who ask me "how can I classify X" or "is there any other tool out there" or "the Pix4D/Metashape/etc. classifier is bad", so there is a LOT of interest in having such a tool available. I know I would immediately incorporate such a tool into my graduate level instruction and would tell pretty much everyone I know to use it, which would generate plenty of citations :)

I wish there were some way that I could help. If you can think of any, please let me know (send me an email at "weidner" at mines.edu).

Here are a couple of ideas:

1. Obviously giving credit is a priority. There are ways to create a DOI for the source code, such as using a service like CERN's Zenodo (https://zenodo.org/) linked to a Github repo.
2. I'd be happy to provide input regarding design of a simplified GUI, testing and/or contributing to writing an understandable documentation.

[Dimitri]

Hi,

thanks for your detailed response, very good ideas for referencing and link to your work (love the stuff on tracking trees to map landslide deformation !).

I've put releasing 3DMASC on my plate. I'll discuss with my colleagues to get a paper out discussing some of the innovative aspects of 3DMASC (joint analysis of several point clouds, + a few innovative features), but also the objective of having a plugin available to anyone and essentially based on a GUI. Being realistic about the time I actually have to work on a paper (I'm a slow writer...), I've definitely noted the possibility to have a zenodo DOI to the plugin, and If I see that I wont' have time for the paper, we'll release it that way.

In the meantime, we'll also try to polish the GUI to have a very simplified version for non-specialists. The current version is still too complex, I don't see myself using it for teaching for instance, the way I can use CANUPO for absolute beginners on points clouds. We now have an engineer in our team able to delve into CC code, and we even have a new plugin fully developped in the hood : ICP_M3C2 (I'm sure you'll have guessed what it does ;-)...but that we'll have to wait after 3DMASC ;-)

When we'll release the 3DMASC plugin, any help in writing tutorials and documentation will be welcomed ! In the meantime if you want to experiment with it, and give us your feedback on how to improve some aspects, send me an email.

[toxicmag]

Dear folks,

i had a quick chat with Luke about this topic and i wanted to drop these lines here to encourage you to continue.
:) By the way: there are a lot more people i know who tried to improve CANUPO and/or coding their own alternatives...
But still there's no succesful result yet. People (like me) are not good enough in programming. Many of them are simply users - not coders.

CANUPO did and still does a good job but it's time for the next step. :)

Let the community know what's the biggest issues currently to develop a good GUI for the CC-nation out there.
I personally - and i bet a lot more people - would try our best to help you with this.

Cheers from Bavaria, Germany.

Alex

[PablerasBCN]

Hello,

Thank you for developing CANUPO, it really helped me to extract ground and retaining walls better than CSF Filter (despite does a great job).

I use public MMS data.

I also tried to train to extract powerlines, althought it adds them to the class, also adds a lot of noise I need to filter manually. Same for other elements like traffic barriers etc. Anyways I'm very happy.

1) I could provide manually segmented data of mobile mapping scan, in case it is anyhow usefull.

2) May be, for the GUI, if it is very hard to sumarize everything in checkboxes, tabs etc, because there may be interdependencies on what is selected and what not etc. A good way could be a node based GUI, where each method is represented by a box with inputs and ouputs. Theese way users can build a complex set of settings. Sort of visual programming or visual command line.

something like this


The gui could be a separate exe, then it could convert the visual part to a string of commands and execute theese into CC or wherever else, this way the UI work can be used with multiple programs.