Better build speed

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Better build speed

jcelerier
Hello,

I added a branch (feature/cotire_2) which adds cotire (https://github.com/sakra/cotire) support to the jamoma build. Basically, cotire is a cmake library which uses various techniques in a cross-platform way to greatly decrease build time; notoriously :
- precompiled headers (they are discovered automatically but there is a lot of option in the cotire manual to tweak this)
- unity build (all the .cpp files of a single target, e.g. a dylib are appended and built together).

For now I put it behind a macro (JAMOMA_USE_COTIRE) so cmake would have to be called with -DJAMOMA_USE_COTIRE:Bool=True (or setting it in the CMake GUI).

PCH is automatic, and unity build, being more dangerous, has to be called manually at build time (e.g. make all_unity or the all_unity target in Xcode, etc...).
For instance the macro #define thisTTClassName in most .cpp files could cause problem with this.

For now I added it only to JamomaCore.
Some numbers, on my desktop machine (i7 2600k, 4ghz, 16gb of RAM) :
- No cotire : make all -j8  892,96s user 45,21s system 720% cpu 2:10,15 total
- Cotire (PCH) : make all -j8  629,53s user 33,55s system 672% cpu 1:38,65 total
- Cotire (Unity) : make all_unity -j8  383,69s user 17,63s system 719% cpu 55,809 total

So unity build actually halves build time, which is, I think, a desirable propriety for continuous integration & deployment since this could mean building more configurations in a single CI run (Travis for instance limits a build to 40 minutes).

For now I only tested under Linux, I'm on my for some OS X tests.

Best regards
Jean-Michaël
ps: I also acted the removal of boost & TTPath which were not used anymore anywhere; I had done this previously in the feature/mingw branch but it was not merged and it has certainly suffered from some level of bit-rot so now it's in dev.



------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel
tap
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Better build speed

tap
Great work!  Thanks on both counts!

Tim


On Fri, Jul 10, 2015 at 2:55 PM, Jean-Michaël Celerier <[hidden email]> wrote:
Hello,

I added a branch (feature/cotire_2) which adds cotire (https://github.com/sakra/cotire) support to the jamoma build. Basically, cotire is a cmake library which uses various techniques in a cross-platform way to greatly decrease build time; notoriously :
- precompiled headers (they are discovered automatically but there is a lot of option in the cotire manual to tweak this)
- unity build (all the .cpp files of a single target, e.g. a dylib are appended and built together).

For now I put it behind a macro (JAMOMA_USE_COTIRE) so cmake would have to be called with -DJAMOMA_USE_COTIRE:Bool=True (or setting it in the CMake GUI).

PCH is automatic, and unity build, being more dangerous, has to be called manually at build time (e.g. make all_unity or the all_unity target in Xcode, etc...).
For instance the macro #define thisTTClassName in most .cpp files could cause problem with this.

For now I added it only to JamomaCore.
Some numbers, on my desktop machine (i7 2600k, 4ghz, 16gb of RAM) :
- No cotire : make all -j8  892,96s user 45,21s system 720% cpu 2:10,15 total
- Cotire (PCH) : make all -j8  629,53s user 33,55s system 672% cpu 1:38,65 total
- Cotire (Unity) : make all_unity -j8  383,69s user 17,63s system 719% cpu 55,809 total

So unity build actually halves build time, which is, I think, a desirable propriety for continuous integration & deployment since this could mean building more configurations in a single CI run (Travis for instance limits a build to 40 minutes).

For now I only tested under Linux, I'm on my for some OS X tests.

Best regards
Jean-Michaël
ps: I also acted the removal of boost & TTPath which were not used anymore anywhere; I had done this previously in the feature/mingw branch but it was not merged and it has certainly suffered from some level of bit-rot so now it's in dev.



------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel



------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Better build speed

Antoine Villeret
Hi, 
good job JMC !
On JamomaPureData side, it's now building on Travis CI in container based infrastructure which is faster than legacy VM.
The whole project (JamomaCore and JamomaPuredata) builds in 11 min 38 sec.
Appart of this reduce build time, the VM starts in less than 10 seconds whereas legacy ones takes usually 1 to 10 minutes to start.
 
And concerning the build timeout on Travis, I think the timeout is 120 minutes but it is a per job timeout [1].
So you can always split the build with a matrix (for example one job per arch) which takes more than 2h to complete.

[1] : https://travis-ci.com/plans under "Are there any restrictions on build time?" section

Le sam. 11 juil. 2015 à 03:32, Timothy Place <[hidden email]> a écrit :
Great work!  Thanks on both counts!

Tim


On Fri, Jul 10, 2015 at 2:55 PM, Jean-Michaël Celerier <[hidden email]> wrote:
Hello,

I added a branch (feature/cotire_2) which adds cotire (https://github.com/sakra/cotire) support to the jamoma build. Basically, cotire is a cmake library which uses various techniques in a cross-platform way to greatly decrease build time; notoriously :
- precompiled headers (they are discovered automatically but there is a lot of option in the cotire manual to tweak this)
- unity build (all the .cpp files of a single target, e.g. a dylib are appended and built together).

For now I put it behind a macro (JAMOMA_USE_COTIRE) so cmake would have to be called with -DJAMOMA_USE_COTIRE:Bool=True (or setting it in the CMake GUI).

PCH is automatic, and unity build, being more dangerous, has to be called manually at build time (e.g. make all_unity or the all_unity target in Xcode, etc...).
For instance the macro #define thisTTClassName in most .cpp files could cause problem with this.

For now I added it only to JamomaCore.
Some numbers, on my desktop machine (i7 2600k, 4ghz, 16gb of RAM) :
- No cotire : make all -j8  892,96s user 45,21s system 720% cpu 2:10,15 total
- Cotire (PCH) : make all -j8  629,53s user 33,55s system 672% cpu 1:38,65 total
- Cotire (Unity) : make all_unity -j8  383,69s user 17,63s system 719% cpu 55,809 total

So unity build actually halves build time, which is, I think, a desirable propriety for continuous integration & deployment since this could mean building more configurations in a single CI run (Travis for instance limits a build to 40 minutes).

For now I only tested under Linux, I'm on my for some OS X tests.

Best regards
Jean-Michaël
ps: I also acted the removal of boost & TTPath which were not used anymore anywhere; I had done this previously in the feature/mingw branch but it was not merged and it has certainly suffered from some level of bit-rot so now it's in dev.



------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel


------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel

------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Better build speed

jcelerier

On Thu, Jul 16, 2015 at 12:02 PM, Antoine Villeret <[hidden email]> wrote:
it's now building on Travis CI in container based infrastructure

Does this mean more recent ubuntu versions at last ? :)
Also did you manage to install all the packages easily ? I'd like to add i-score to the container infra but it requires Qt5.4 and I have to fetch it from a ppa (as well as sudo installation of recent gcc, cmake).


Best
Jean-Michaël

------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Better build speed

Antoine Villeret
2015-07-16 12:14 GMT+02:00 Jean-Michaël Celerier <[hidden email]>:

On Thu, Jul 16, 2015 at 12:02 PM, Antoine Villeret <[hidden email]> wrote:
it's now building on Travis CI in container based infrastructure

Does this mean more recent ubuntu versions at last ? :)
unfortunately not, it seems to be on the road, but not ready yet
 
Also did you manage to install all the packages easily ? I'd like to add i-score to the container infra but it requires Qt5.4 and I have to fetch it from a ppa (as well as sudo installation of recent gcc, cmake).
yes, you can't use sudo but you can add ppa and package to install to your .travis.yml
and you can always download binary package with wget, put them in user space and use them as is.
this is what i'm doing with CMake for example

best 

a

Best
Jean-Michaël

------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel



------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Better build speed

Antoine Villeret
Hi, 

I'm trying to setup CI with cotire and unity build.
Cotire runs fine 13% faster.
Unity build runs fine on Linux and Rpi.
It's 86% faster on RPi but 15% slower on Ubuntu.
Unity build fail on OS X, see the report : https://travis-ci.org/jamoma/JamomaCore/jobs/78616348
Jean-Michaël did you manage to successfully make a unity build on OS X ?

a+

Antoine

--
do it yourself                      
http://antoine.villeret.free.fr

2015-07-16 12:30 GMT+02:00 Antoine Villeret <[hidden email]>:
2015-07-16 12:14 GMT+02:00 Jean-Michaël Celerier <[hidden email]>:

On Thu, Jul 16, 2015 at 12:02 PM, Antoine Villeret <[hidden email]> wrote:
it's now building on Travis CI in container based infrastructure

Does this mean more recent ubuntu versions at last ? :)
unfortunately not, it seems to be on the road, but not ready yet
 
Also did you manage to install all the packages easily ? I'd like to add i-score to the container infra but it requires Qt5.4 and I have to fetch it from a ppa (as well as sudo installation of recent gcc, cmake).
yes, you can't use sudo but you can add ppa and package to install to your .travis.yml
and you can always download binary package with wget, put them in user space and use them as is.
this is what i'm doing with CMake for example

best 

a

Best
Jean-Michaël

------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel




------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Jamoma-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jamoma-devel
Loading...