L. Arnold's picture

Moved from my Blog Comments on:

-------------
post 1
-------------

I am working on sequencing of TKLDEV and how to land a GIT package.

----------------------

I am working on sequencing of how to setup a forked App.

(WHAT does this README.rst do besides give documentation?)

root@tkldev fab/products# wc -l $(find core -type f|grep -v '\.git\|png\|jpg\|changelog\|README.rst\|txt$')

  3 core/Makefile
  1 core/plan/main
  2 core/conf.d/main
  6 total

However in more complicated Lamp etc below:

Again the Readme.rst - is it giving any sequencing information?

root@tkldev fab/products$ wc -l $(find lamp -type f|grep -v '\.git\|png\|jpg\|changelog\|README.rst\|txt$')


  11 lamp/plan/main
   1 lamp/overlay/etc/skel/.bashrc.d/php-xdebug
  17 lamp/overlay/etc/apache2/mods-available/status.conf
  23 lamp/overlay/etc/apache2/sites-available/default
  10 lamp/overlay/var/www/cgi-bin/test.cgi
  73 lamp/overlay/var/www/index.php
   1 lamp/overlay/var/www/phpinfo.php
   5 lamp/conf.d/main
   1 lamp/removelist
   5 lamp/Makefile
 147 total

So I am seeing that the Overlays, are going in AFTER MAIN, and Creating Folders to Load the CONF.D /MAIN file into.  (not yet clear about removelist)

---

So Finally, Why Does the LAPP (app) not have a "conf.d" folder or "main" file in it?  I was wanting to try a Mod of Lapp.  Want to compare that set up to on w/ Nginx and one with simple Postresql  (adding Odoo on top)

I will also add my question just posted on Tracker.  Why does the Conf.d/main in the LAPP app call Lapp itself?  I would think it should perhaps call Postgresql, then add Apache/SSL to it.  I have more questions but will leave them from here.

Thanks for the post above.  It is helpful.  Somewhat daunting.  Sequence and simplicity of understanding on my side is my goal just now.

will follow w/ 2 more comments/questions
Forum: 
Tags: 
L. Arnold's picture

Found these details

Walkthroughs helping but need more clarification if possible.

https://github.com/turnkeylinux-apps/tkldev/tree/master/docs

https://github.com/turnkeylinux-apps/tkldev/blob/master/docs/development/walkthrough.rst

Working right now on how to AIM and Land a GIT Folder or a Packaged installer into TKLDEV

In general I am wanting to push the following into a TKLDEV ISO

https://www.odoo.com/documentation/8.0/setup/install.html

  Assuming one wants to update to Latest Version Software frequently, does it matter if fed by git or wget?  How to find a stable build so the target stops changing.  (quite irritating this was moved from BZR recently)

some usable info on these two links:

http://pulse7.net/openerp/install-odoo-ubuntu-14-04-github/

https://matmoz.input.center/blog/our-news-1/post/odoo-8-on-debian-7-1

Moved over from Blog Comment to Forums (by me).

Will do a hand build next but this seems like it should be easy w/ TKLDEV.  Just not understanding some parts still.

Thanks in advance.

 

L. Arnold's picture

I just ran into these two page which seems quite practical for building openerp/odoo

http://pythonhosted.org/anybox.recipe.openerp/ 

https://pypi.python.org/pypi/anybox.recipe.odoo

They could be helpful in building TKLDEV image.

Jeremy Davis's picture

What does this README.rst do besides give documentation?

Generally a README is just a text file for info. Although in the case of the TurnKey appliances, you will note that it is actually the text for the appliance page on the website (e.g. compare https://github.com/turnkeylinux-apps/core/blob/master/README.rst with the text on http://www.turnkeylinux.org/core). The appliances pages are auto built from the info contained on GitHub...

TBH I am not quite sure what your command is trying to do. AFAIK 'wc -l' is just counting newlines in files ['wc' is wordcount; -l switch is lines...] so I don't understand how that would have any further meaning...?! But perhaps I've missed something?

Again the Readme.rst - is it giving any sequencing information?
...
So I am seeing that the Overlays, are going in AFTER MAIN, and Creating Folders to Load the CONF.D /MAIN file into. (not yet clear about removelist)

As per my comments above; I am unclear on how you reach a conclusion that the command you are using is giving you info about order of execution... In my experience, this is the order in which stuff is applied:

  1. install packages from 'plan/'
  2. overlay files from 'overlay/'
  3. run scripts in 'conf.d/'

removelist just removes anything that was required for build but not required for the appliance to run (i.e. a way to remove stuff that was only required temporarily) but TBH I've never used it...

Why Does the LAPP (app) not have a "conf.d" folder or "main" file in it?

LAPP doesn't have a conf.d directory because it needs no further configuration (beyond the packages installed (from the plan) and the files included in the overlay. If you were to base a new appliance on LAPP you would just add the conf.d directory and put your conf script(s) in it...

Why does the Conf.d/main in the LAPP app call Lapp itself?

I've edited this answer because I think I worked out what you were asking... I assume that you mean lapp/plans/main (LAPP appliance doesn't have a conf.d as discussed above).

To allow recycling of code most of the files that are reused (by multiple appliances) reside in common. This allows a single codebase to be maintained rather than having to maintain the same code in multiple places. All the base appliances (e.g. LAMP, LAPP, etc) have code in common. Shared plans are launched within the plans/main; e.g. for LAPP this line uses the base plan: common/plans/turnkey/base and this line uses the lapp (common) plan: /common/plans/turnkey/lapp. Other code (e.g. common conf files and/or common Makefiles) are launched by the Makefile.

Working right now on how to AIM and Land a GIT Folder or a Packaged installer into TKLDEV

It doesn't really matter where the source comes from; but for an official TurnKey appliance it is important that it has a stable version of the software. Generally that is not an issue if you just wget an archive (it will usually be a specific version; or sometimes just the latest stable version). With git though; more often than not the 'master' branch will be a 'work in progress' type branch (or even a development branch) - almost always it should build and work though. Usually software managed by git will have stable versions tagged (with a version number pinned to a specific commit) and/or have release archives (on GitHub, as of relatively recently tagged commits are automatically available as zipped releases).

FWIW my suggested workflow is documented here.

L. Arnold's picture

Thanks for your help.  I will review in detail.

Jeremy Davis's picture

I just added some extra info to my post above. I now have a much deeper understanding of how TKLDev works under the hood and I think I get what you were asking, so I thought I'd add some more detail... All the extra stuff is under
Why does the Conf.d/main in the LAPP app call Lapp itself?

As I wrote above, I get that you probably meant lapp/plans/main

L. Arnold's picture

I am not always clear but I do try to parse outlines from threads.  I will look in detail at your http://www.turnkeylinux.org/docs/tkldev-new-appliance instructions  (expect I was there before).

I have to look at the LAPP plan again, but I do recall it's calling itself  (that perplexed me)... In Lapp/ Plan/Main it was calling right from the start ...   (you are right, it is not in conf.d).  

#include <turnkey/base>

#include <turnkey/lapp>

I believe you are saying that the turnkey/lapp call is different than the Turnkey Lapp (App build in GitHub)  


So on to my objective?

Did you build a Fork of Lapp for Odoo?  Will see if I can find one.  I'm on the fence between Apache and NGINX for Odoo.  I have seen recent call outs for Apache but most have been for NGINX.  Apache I can navigate so that pulls me that way.  So far my Odoo homebrew is in Postgresql App, so I am comparing different App builds to see how they come together.

In my last round on this a few weeks ago I was basically setting up a horse-race.  Three approaches were modifying the Postgresql, the LAPP, or the Tendenci appliance to build Odoo.   Doing so they had very different structures though and I was therefore trying to understand the sequencing system of TKLDEV. 

Less chat, more do.  I will read more on all this.

Jeremy Davis's picture

But Apache is more common (the most used open source webserver), has much more documentation and tutorials to configure and is the only open source webserver with a Webmin module. So as it is currently not realistic for TurnKey to provide appliances with multiple bases this will need to be a LAPP based appliance to make it into the library. Moving forward there may be possibilities for optional bases (such as Nginx) but we'll have to wait and see as it won't happen for v14.0. So at thos point all TKL php and python webapp appliances are built on top of LAMP or LAPP...

So like it says in the instructions; clone a copy of the LAPP repo to your TKLDev server (don't fork it); then rename it and reinitialise it (so it's no longer a clone of LAPP, but it's own appliance). I'm sure it's all documented in TKLDev docs...

Jeremy Davis's picture

And it looks like I didn't commit anything. From memory I just played in the TKLDev root.tmp and used that to figure it out and test but never got as far as committing anything so there is nothing to show beyond the instructions and notes in that other thread...
L. Arnold's picture

  1. I've installed TKLDEV and gotten running.
  2. Made a GitHub Repository where I cloned LAPP appliance to my own.
  3. Pulled into my TKLDEV Machine the GitHub Repository 
  4. Renamed it to the new APP Name
  5. I have not changed anything else.
  6. Got all the way to Line 6 in your Blog Post..
  7.  Thought I could do a test Make but

When I run Make as recomended in your blog post  at (root/new-app-name) or in in  /turnkey/fab/products /new-app-name (as directed by https://github.com/turnkeylinux-apps/tkldev/blob/master/docs/setup.rst)

I get the errors together (from either folder)

/turnkey/fab/common/mk/turnkey/new-app-name.mk:  No such file or directory

/turnkey/fab/common/mk/turnkey.mk:  No such file or directory

make: *** No rule to make target '/turnkey/fab/common/mk/turnkey.mk'

Thoughts on how to rectify?  There is no "common" folder whatsoever in /turnkey/fab/.  Only /bootstraps and /products folders there.

(new-app-name = odoo-lapp)

I am guessing the information is in this Doc - though all I changed was the name -- and it is in my repository not TKLs:  https://github.com/turnkeylinux-apps/tkldev/blob/master/docs/helloworld.rst

Jeremy Davis's picture

I overlooked spelling that out in the post, but until we release the new improved version of TKLDev is released you still need to manually do the setup. Check the TKLDev docs here.

Also I should perhaps remove that mention of adjusting the Makefile? You don't need to add anything there unless you wish to add something from common. And my guess is that most people who have made a sane base appliance decision won't need (or want) to make any changes there...

L. Arnold's picture

It seeme that tkl-dev setup did all the things that are spelled out in the docs below that step under "manual install".  Everything seemed to work.

The Docs do have a "manual" component below the seemingly "one step" tkl-dev install process.  (Not seeing th link as I type here but the referece should be in one of the comments directly above).

L. Arnold's picture

I had looked and looked but missed the subfolder "turnkey"

https://github.com/turnkeylinux/common/blob/master/plans/turnkey/lapp

I have to admit, "blob/master" isn't really obvious when you navigate GitHub either.

This is helpful to see what really is dropping in.

Moving on now.  (to the bottom for process)

L. Arnold's picture

ISSUE SOLVED via COMMENT BELOW this ENTRY

to see how I got there read on

Yesterday:  I Cloned Lapp, renamed to Odoo-lapp.  Briefly changed the line

include $(FAB_PATH)/common/mk/turnkey/lapp.mk

to:

include $(FAB_PATH)/common/mk/turnkey/odoo-lapp.mk

(Changed it back shortly thereafter but after my Pull to the Turnkey Directory.  When I repulled to the Root version directory I verified the change and assumed all was good.  It was not.  This caused my error in that I changed it back but did not pull to the correct directory as was following 2 sets of instructions and had 2 sets of code in TKLDEV)

Today:

ran tkldev-setup

 Ran Make

Error Reads:

Makefile:4: /turnkey/fab/common/mk/turnkey/odoo-lapp.mk:  No such file or directory

make: *** No rule to make target '/turnkey/fab/common/mk/turnkey/odoo-lapp.mk".

Stop

----------

It noticed after TKLDEV-SETUP that a Core folder had been brought in.  I did not Clone it myself.

Changed to Folder Core (from odoo-lapp which is parrallel)

Ran MAKE on CORE:  ran successfully.

------------

Since My Odoo-Lapp does not install however (cloned, renamed but otherwise unchanged LAPP) ???

Perhaps since I changed the name of Lapp to Odoo-Lapp perhaps I need to help new folders withthat name be created that are not crated by the General Make Process.

If I had this still be Lapp, would it run by itself?

I am missing something.

-----------

Postscripts Below 

-----------

1:  potential issue: proceeding but  can I not run this from a OVF Built system?  You reference OVZ problems here:

http://www.turnkeylinux.org/forum/support/20140824/trying-install-minecraft-turnkey-server

that Core make is working it would seem the system is operating so red flag.

-----------------

2: TKLDEV-SETUP:  Yesterday at TKLDEV installation, this didn't jump out of the blog instructions I was following in the blog post (Docs are good) but

https://github.com/turnkeylinux-apps/tkldev/blob/master/docs/setup.rst

./tkldev-setup

B. Setup the build dependencies

In the next version of TKLDev, you can skip this step because it happens automatically on first boot.

If it doesn't (e.g., we didn't have a working Internet connection) you'll be able to trigger it by running this command inside TKLDev as root:

But you're not using the next version of TKLDev (yet), so you'll want to download the new tkldev-setup from GitHub:

cd /usr/local/sbin
wget https://raw.githubusercontent.com/turnkeylinux-apps/tkldev/master/overlay/usr/local/sbin/tkldev-setup
chmod +x tkldev-setup

./tkldev-setup

Do the "Manual Installs" also need to be done after /tkldev-setup ?  They appear to have been built w/ tkldev-setup

The build dependencies: what are they?
  1. bootstrap in /turnkey/fab/bootstraps: contains minimal bootstrap filesystems.
  2. cdroots in /turnkey/fab/cdroots: contains the cdroot template for the built ISO.
  3. common in /turnkey/fab/common: contains source code shared amongst all TurnKey systems.

It appears this was all built in TKL-Dev-setup command

That Core built it would appear all is OK for the system.  

I will look at Pulling Lapp in by itself and seeing if it will Make by itself.  Then it will for sure be something w/ the Name Change I gave yesterday, from your docs/blog post:

---------------------

(My PROBLEM was not navigating to /turnkey/fab/products/new-app-name folder before running these commands.  Therefore dropped separate file set into Root/new-app-name

However, I also followed the GitHub docs and did pull the folder there - just had a test error in the MakeFile which is explained in the comment below)

git clone https://github.com/turnkeylinux-apps/<base-appliance-name>.git
mv <base-appliance-name> <new-appliance-name>
cd <new-appliance-name>
rm -rf .git
git-init
L. Arnold's picture

The Blog/Docs Post is pulling a folder into /root

The GitHub Docs is pulling into /turnkey/fab/products

Yesterday I had "as a test" temporarily changed the MakeFile Line

include $(FAB_PATH)/common/mk/turnkey/lapp.mk

to

include $(FAB_PATH)/common/mk/turnkey/odoo-lapp.mk

And when that was in place did a Pull to the /turnkey/fab/products/odoo-lapp folder

I changed the line back to default.  Pulled (git pull origin master) on root/odoo-lapp  (verified the change) and thought all was good.

TODAY/JUST NOW

I went again to (via CD) /turnkey/fab/products/odoo-lapp

ran

git pull origin master

(1 change - from yesterday noted in the pull)

ran

Make

MAKE appears to be running Now!  (Did so Successfully it seems, now to find the ISO?)

(I did see a some warning Table Mounted Directories but appears still to be proceeding)

Takes a lot longer to build LAPP than CORE I am noting.

Jeremy Davis's picture

Like I said above, I think that I probably should remove the reference to adjusting the makefile. Generally you won't want to change that.

So if you haven't already change:

include $(FAB_PATH)/common/mk/turnkey/odoo-lapp.mk

back to

include $(FAB_PATH)/common/mk/turnkey/lapp.mk

(which I now read you have already...)

Ran MAKE on CORE: ran successfully.

Excellent, that proves that your environment is set up properly... So personally I would start again...

If you still have a odoo directory in /turnkey/fab/products/ then move it:

cd /turnkey/fab/products/
mv odoo odoo-old

Then clone LAPP again. And start again...

cd /turnkey/fab/products/
git-clone https://github.com/turnkeylinux-apps/lapp.git
mv lapp odoo
cd odoo
rm -rf .git
git-init
git-add . #note the dot!
git-commit -m "initial commit of tkl lapp appliance"

Now assuming that you followed the steps to set up your GitHub account (in the wiki post I did), you can now link your new repo to GitHub. Go to GitHub & sign in. If you have already created an odoo repo there, please delete it. That will require going to the GitHub repo in your browser and on the right hand side (about half way down), click settings. On the setting page, scroll down to the bottom and click 'delete this repository' and follow the prompts.

Now (still on GitHub) create a new clean repo. GitHub should lead you through it, but if you're not sure they have a good help page for making a new repo: https://help.github.com/articles/creating-a-new-repository/

Now you want to link the new blank repo on GitHub to your local TKLDev lapp repo (which you have renamed odoo). IIRC GitHub should already help you out with that, but just in case, they mavbe a help page for that too: https://help.github.com/articles/adding-an-existing-project-to-github-us...

Once all that's done, check it works by pushing your odoo repo (which is still just vanilla LAPP really) to GitHub. Do that like this:

git push

Now the next step I would take would be to build the vanilla LAPP appliance (which you have renamed odoo just to make sure everything works...

cd /turnkey/fab/products/odoo
make CHROOT_ONLY=y
L. Arnold's picture

I believe that I have made a new Repo (my name is odoo-lapp) and it is a vanilla lapp copy.  It is not a fork but is something that is in my control right now.

I think you were seeing me hung up and restarting the project.  However, never sure.  Not sure that I ran a Git Push or not, though my memory says yes (been 5 days since).  Will check the notes.

"once that is done try pushing" you write..  got it.

No matter, this is a good summary for how to make a fresh clean repo based upon another.

---------

Quick Question though.

So having run Make, seemingly successfully, where does the ISO land?  Do I need to clean out any of the "Made Files" before I run Make again?  Does it recycle or make a series of new folders and ISO images?

I will read up.  Start of the session just now.

Jeremy Davis's picture

If you want to rebuild from scratch then run 'make clean' first.
L. Arnold's picture

I just organized all my Browser Tabs on the Current Subject.  Have not really started changing the App yet, but to get to Here (2 Makes, one for Core, one for Lapp)  I have 25 Tabs OPEN. (will start closing)

One difficulty I see is that often instructions refer to other instructions and, in the short rabbit run just now, one or two small details got left off the follower's (my) outline.

Important details are mostly in the TKLDEV-GitHub Docs, but trying to follow the TKLDOC/Blog entry is attractive because Several pages are condensed to one.

Now my notation needs to be updated to be sure today's Tkldev-setup and be sure the Folder path is in Turnkey/fab/products.

Excited to start building the APP now.

Jeremy Davis's picture

Sounds like you got it in the end...

Great work though that you have things underway! :)

L. Arnold's picture

Assuming that LAPP will be modified in the future and that at present we are discussing a "copy" of LAPP as today, would it not make sense to have a completely Bare system which simply includes whatever is in the current version of LAPP, and then goes on to modify it?

 I can see that the TurnkeyLinux-apps/lapp is indeed quite Bare (and including the Base and Lapp plan in 

https://github.com/turnkeylinux/common/tree/master/plans/turnkey

----------

But I can see that many Apps do not have a Plan under the common tree.  Lets take Magento for a moment.  Lets say I wanted a Magento App that is based upon Magento but which has significant changes.  I would want to take ongoing changes to the Magento App, but I would also want to have 3 other downloads go in, and when the version changes, the general capability of the system would change.  Moreover, if there were major changes to the Magento-App those would be automatically pulled and integrated to the changes I wanted to have.

Esoteric process question perhaps.

Jeremy Davis's picture

Although the likelihood of us creating multiple servers that are based on an appliance like Magento are unlikely; but if there were; then there would be magento build info in common...
L. Arnold's picture

https://github.com/turnkeylinux-apps/tkldev/blob/master/docs/tips.rst

Brings some clarity to this.

(Reference of following comand I want to understand more.

make clean && Make

Likewise.  What would be the command to rerun Make but not Install any Packages OR Downloads?

(I.E.  I am just working on the config files coordinating correctly)

Jeremy Davis's picture

When you run make it follows the instructions in the Makefile; which is designed to install all the relevant plans, overlay all the overlays and run all the relevant conf.d scripts. So there is no way to make without doing that stuff...

Having said that you can undo the last step of a complete build by undecking build/root.sandbox and removing the build/stamp/root.sandbox

deck -D build/root.sandbox
rm build/stamp/root.sandbox

You can also do that to root.patched too; that makes it much quicker to redo...

L. Arnold's picture

Hopefully my learning is shareable.

Really like what a I have been able to build.  Will ask other questions separately.  Simple is the kay.  

One last ?  If I change one .conf or .php file on GitHub, then run git pull..  do I have to run make clean, or can it just be make?

I would love to see some git and package repositories that can be declared as "built" so that so much time is not burned pulling both.  Make will use pre pulled packages especially helpful with abtimeout failure.  It does not work with Git.  I am not sure what happens with small files. Perhaps they integrated in subsequent makes?

I will work on understanding and testing your last comment jeremy.  Likely a trove of info that just needs to be digested with some practice.

Do it well and the issues go away.

Thanks to all.

Jeremy Davis's picture

So you can do a full new clean (as you already know) like this:
make clean
make
Which will basically wipe the whole thing clean and start again from scratch. But as I hinted in my comment above, you can just rebuild the last couple of steps which will usually be much quicker... Do that like this:
deck -D build/root.sandbox
rm build/stamp/root.sandbox
rm build/stamp/root.patched
make
So long as your last build was successful that should work everytime. If your last build failed though, sometimes you need to kill stray processes in your root.patched deck. Do it like this:
fuser -k build/root.patched
Great to see you having so much fun with it all! :)
Ken Robinson's picture

And found this thread...  Can learn a lot from browsing the forums!

What does the below do? Can it be run on root.patch? What I do now is run top and kill processes manually.

 

fuser -k build/root.patched

Regards,

Ken  
":0)

http://www.github.com/DocCyblade

Jeremy Davis's picture

and the k switch force kills them.

So

fuser -k build/root.patched

finds all the process PIDs which are accessing files within the build/root.patched dir and kills them...! :)

Ken Robinson's picture

Work smarter not harder! duh!

f user / file user - makes sense now...

Regards,

Ken  
":0)

http://www.github.com/DocCyblade

Add new comment