Difference between revisions of "ABF Console Client"

From Rosalab Wiki
Jump to: navigation, search
(Rework page)
Line 1: Line 1:
 
= Intro =
 
= Intro =
−
Let's take a look at package life-cycle. User wants to create (or get existing) project, modify it, build and publish. Let's start in order.
+
ABF Console Client aims to simplify remote work with ABF using command line and supports most active stages of a project life cycle - modification, building and publishing.
  
−
# '''Clone git repository. ''' A project name is assumed to be known. User have to open ABF web page, search for a project, go to a project page, copy git URL, clone repository using git. However, it can be done easier. Just execute "abf get PROJECT", where PROJECT is a project name with owner (owner/proj_name). Owner can be omitted, default group will be used.
+
= Getting Started =
−
# ''' Project modification. '''  When you've got files in your project modified, you can execute "abf put -m MSG". It will execute "git add --all", "git commit -m MSG", "git push". Additionally, all the source archives have to be uploaded to File-Store and .abf.yml file have to modified. It's a routine work, especially when you have to build a hundred projects a day. Console client will do it for you automatically.
+
Typical actions with ABF console client can be performed in the following way:
−
# ''' Create a build-task. ''' To do it, you have to open an ABF web page, click some check-boxes and click 'Start build'. It looks like a few work to do. But if you have to build hundreds packages a day, you can do it much faster using console client, especially if you write your own script executing console client.
+
−
# ''' Building process. ''' To check the current building status, you don't have to check web page anymore. Just execute "abf buildstatus ID" to get a short summary. ID can be omitted and the status for the last built project. It will be described later in detail.
+
−
# ''' Publication. ''' "abf publish ID" will do it easily.
+
  
−
As you can see, console client makes maintainer's work easier a bit.
+
* '''Clone project Git repository '''  
  
−
The list of some features:
+
abf get PROJECT
  
−
* ABF client caches the locations of git-repositories in your system when it accesses them. If you have already got lots of repositories, you can cache their locations at one go. Just run "abf locate update-recursive -d PATH", where PATH is a directory with repositories. For what perpose should one know the location of every repository? For example, user can now execute "abfcd PROJECT". You can just learn the location of any project by executing "abf locate -p PROJECT". Console client will be able to move files not only among the project branches, but also among projects.  
+
where PROJECT is a project name with owner (owner/proj_name). Owner can be omitted, default group will be used. This command is equivalent of "git clone" invoked with URL to corresponding Git repository.
  
−
* First version of console client had only got the basic version of bash autocomplete script. Now it autocompletes almost everything: option names, git branches, build and save-to repositories for "abf build" (the latter requires a project name to be specified, so you have to type the --project/-p before --save-to-repository/-s ). As a result, it's much easier to work with console client, because you don't have to type the long repository name or remember the list of possible variants. Sometimes autocompletion works long (about 1 second), but the results are getting cached and the process is speeding up.
+
* ''' Commit local project modifications and push them to Git repository '''
  
−
* The new command, "abf clean" appeared. It scans the spec and .abf.yml files and current directory and shows problems found. If some source or patch file can not be found (or resolved via URL or .abf.yml) - it prints an error message. It also prints warnings, for example, if a file is specified in .abf.yml and in spec file. The test is automatically executed while creating a build task from the current directory and prevents a build task from being sent. This check can be turned off by option --skip-spec-check.
+
abf put -m MSG
  
−
* When user creates a new build task, its ID will be stored and associated with a project. "abf buildstatus" will print the summary for the last build task. If project was specified (or you are in a project directory) - it will print information about the last build task for this project.
+
This command first scans the directory for binary files (e.g., tarballs with source code), upload them to ABF File Store and put their hash sums into {{file|.abf.yml}} file. Then it parses spec file and tries to detect Sources which are present in .abf.yml but are no longer used. If these sources are binary files mentioned in .abf.yml, then they are moved to the "removed sources" section of that file so they will not be fetched from File Store when package is being built. Then "abf put" invokes the "git add --all; git commit -m MSG; "git push" sequence of commands.
  
−
* About work with API. The number of unnecessary API calls have been significantly decreased. The fact is that when you, for example, requests information about repository, API returns some particular information about platform too. The older version could only use the ID of this platform and downloaded the full information about platform by one more API call. Now this information is stored in platform object that is marked as 'stub'. When you try to access a field that is not loaded but have to present in this class - a new API call will be made and information will be retrieved. As a result, the first console client call doesn't results in dozens of API calls anymore.
+
# ''' Initiate a build '''  
  
−
* One more interesting feature - work with (http://en.wikipedia.org/wiki/HTTP_ETag) to cache the results of API calls with autovalidation. It results in speeding the process up without a risk of working with the outdated data. ABF is not well configured now to fully support this technology, but it's going to process requests for cached data fast. It will also stop increasing a counter of API calls (every IP and login are limited to 2000 API calls per hour).
+
abf build
  
 +
When called without arguments, this command takes a look at active Git branch, asks ABF about repositories this branch is associated with and initiates a build from the branch to that repositories. You can specify branch to be used and target repositories as additional parameters.
  
−
= Workflow Examples =
+
# ''' Check build status '''
  
−
== Update package ==
+
abf status ID
−
You want to clone a project import/gcc, modify something (modify source tarballs and/or spec) and rebuild the packages.
+
 
 +
This command will provide you with a short information about a particular build task. ID of the task is printed to STDOUT by "abf build". You can omit this parameter; in this case status of the last launched project will be checked.
 +
 
 +
# ''' Publish build task to repository '''
 +
 +
abf publish ID
 +
 
 +
If you package has been successfully built, you can publish it to repository. Alternatively, you can specify "--auto-publish" option when invoking "abf build". Note that the latter option will work only if automated publishing is allowed by repository settings and if there is no package with the same name, version and released already published to the target repository.
 +
 
 +
= Example =
 +
Let's clone import/gcc project, modify something inside it (source tarballs and/or spec) and rebuild the package.
 +
 
 +
* clone a git repository an enter its folder
  
−
  # clone a git repository
 
 
   abf get import/gcc -b rosa2012.1
 
   abf get import/gcc -b rosa2012.1
 
   cd gcc
 
   cd gcc
−
  <some actions with tarballs and spec files>
+
 
 +
*  <some actions with tarballs and spec files>
 
    
 
    
−
  # upload source files to File-Store,  
+
* upload source files to File-Store, modify .abf.yml, commit changes with a given message and push everything to repositories
−
  # modify .abf.yml and commit changes
+
 
−
  # with comment given and push
+
abf put -m "My brand new gcc version"
−
  abf put -m "My brand new gcc version"
+
 
    
 
    
−
  # now you can start a new build task on ABF.
+
* start a new build task on ABF
−
  # all the arguments will be resolved automatically.
+
 
−
  abf build
+
abf build
−
  # it prints IDs of build tasks that were sent.
+
 
−
  # These IDs are also saved in ~/.abf_projects
+
(this command prints IDs of initiated build tasks. These IDs are also saved in ~/.abf_projects)
 
    
 
    
−
  # check a status of your build task.
+
* check a status of your build task
−
  # This command will print statuses for last  
+
** This command will print statuses for the last build task:
−
  # build tasks:
+
abf status
−
  abf status
+
  
−
  # or you can specify IDs explicitly:
+
** or you can specify task IDs explicitly:
−
  abf status <ID1> <ID2> ...
+
 
 +
abf status <ID1> <ID2> ...
 
    
 
    
−
  # if the task has been built successfully,
+
* if the built has been completed successfully, you can publish it:
−
  # you can try to publish it:
+
−
  abf publish <ID1> <ID2> ...
+
  
−
= Command List =
+
abf publish <ID1> <ID2> ...
 +
 
 +
= Command Reference =
 +
Below is the list of all commands officially supported in ABF console client. You can also invoke '''abf --help'' and use '''abf help''' command to get descriptions of the commands and actions supported by your version of the client.
  
 
==help==
 
==help==
−
If you want to get a help for some command - you've got two options: '''abf help <command>''' and '''abf <command> -h/--help'''  
+
If you want to get a help for some command, you can launch either '''abf help <command>''' or '''abf <command> -h/--help'''  
  
 
Options:
 
Options:
  
−
''abf hel <command_name>''
+
''abf help <command_name>''
 
* '''<command_name>''': A command to get help for.
 
* '''<command_name>''': A command to get help for.
 +
 
==alias==
 
==alias==
 
Manage aliases for console client. With aliases you can make your frequently used commands shorter.
 
Manage aliases for console client. With aliases you can make your frequently used commands shorter.
Line 94: Line 105:
 
==== remove ====
 
==== remove ====
 
Remove an alias. The only option is an alias name.
 
Remove an alias. The only option is an alias name.
 +
 +
==clean==
 +
Scan the spec and .abf.yml files for possible problems. First, it detects missing sources or patches (which are mentioned in spec using non-remote path, but are absent in current directory and .abf.yml). The check will also fail if there are some problems with the spec file itself and RPM can't parse it. These checks are automatically executed when creating a build task from the current directory. If they fail, the console client refuses to create a build task. This behavior can be turned off by --skip-spec-check option for the '''build''' command.
 +
 +
In addition, the 'clean' check reports warnings about files specified in .abf.yml and at the same time in the spec file with remote URL. These warnings are not considered to be crucial and don't prevent the build from being started
  
 
==get==
 
==get==
Line 302: Line 318:
 
==test==
 
==test==
 
Execute a set of internal datamodel tests.
 
Execute a set of internal datamodel tests.
 +
 +
= Additional features =
 +
 +
== Cache of local project locations ==
 +
ABF client remembers locations of all repositories ever cloned using it and you can easily move to particular project directory in your local filesystem using "abfcd" command:
 +
 +
abfcd import/abf-console-client
 +
 +
This information is stored in .abf_projects file in your home directory.
 +
 +
You can learn the location of any project in your filesystem by executing
 +
 
 +
abf locate -p PROJECT
 +
 +
If you already have a set of repositories not known to the console client (e,g,m they were cloned using git directly or you removed your .abf_projects file), you can ask the console client to scan particular folders and add all project it finds there to its local cache:
 +
 +
abf locate update-recursive -d PATH
 +
 +
where PATH is a directory with repositories.
 +
 +
Note also that information from .abf_projects file is used by the console client to move files among different projects.
 +
 +
== Bash autocompletion ==
 +
ABF console client support rich autocompletion in Bash. In particular, it can suggest completions for option names, git branches, build- and save-to repositories for "abf build" and a lot of other parameters. As a result, it's much easier to work with console client, because you don't have to type the long repository name or remember the list of possible variants. In some cases autocompletion may take about 1 second, but remember that its results are always cached, so the next time it will be much more faster.
 +
  
 
[[Category:ABF Build Environment]]
 
[[Category:ABF Build Environment]]

Revision as of 11:55, 20 February 2014

Intro

ABF Console Client aims to simplify remote work with ABF using command line and supports most active stages of a project life cycle - modification, building and publishing.

Getting Started

Typical actions with ABF console client can be performed in the following way:

  • Clone project Git repository 
abf get PROJECT

where PROJECT is a project name with owner (owner/proj_name). Owner can be omitted, default group will be used. This command is equivalent of "git clone" invoked with URL to corresponding Git repository.

  • Commit local project modifications and push them to Git repository 
abf put -m MSG

This command first scans the directory for binary files (e.g., tarballs with source code), upload them to ABF File Store and put their hash sums into .abf.yml file. Then it parses spec file and tries to detect Sources which are present in .abf.yml but are no longer used. If these sources are binary files mentioned in .abf.yml, then they are moved to the "removed sources" section of that file so they will not be fetched from File Store when package is being built. Then "abf put" invokes the "git add --all; git commit -m MSG; "git push" sequence of commands.

  1. Initiate a build 
abf build

When called without arguments, this command takes a look at active Git branch, asks ABF about repositories this branch is associated with and initiates a build from the branch to that repositories. You can specify branch to be used and target repositories as additional parameters.

  1. Check build status 
abf status ID

This command will provide you with a short information about a particular build task. ID of the task is printed to STDOUT by "abf build". You can omit this parameter; in this case status of the last launched project will be checked.

  1. Publish build task to repository 
abf publish ID

If you package has been successfully built, you can publish it to repository. Alternatively, you can specify "--auto-publish" option when invoking "abf build". Note that the latter option will work only if automated publishing is allowed by repository settings and if there is no package with the same name, version and released already published to the target repository.

Example

Let's clone import/gcc project, modify something inside it (source tarballs and/or spec) and rebuild the package.

  • clone a git repository an enter its folder
 abf get import/gcc -b rosa2012.1
 cd gcc
  • <some actions with tarballs and spec files>
  • upload source files to File-Store, modify .abf.yml, commit changes with a given message and push everything to repositories
abf put -m "My brand new gcc version"
  • start a new build task on ABF
abf build

(this command prints IDs of initiated build tasks. These IDs are also saved in ~/.abf_projects)

  • check a status of your build task
    • This command will print statuses for the last build task:
abf status
    • or you can specify task IDs explicitly:
abf status <ID1> <ID2> ...
  • if the built has been completed successfully, you can publish it:
abf publish <ID1> <ID2> ...

Command Reference

Below is the list of all commands officially supported in ABF console client. You can also invoke abf --help and use abf help' command to get descriptions of the commands and actions supported by your version of the client.

help

If you want to get a help for some command, you can launch either abf help <command> or abf <command> -h/--help

Options:

abf help <command_name>

  • <command_name>: A command to get help for.

alias

Manage aliases for console client. With aliases you can make your frequently used commands shorter.

For example, you are going to work only with packages for rosa2012.1, you have to checkout to this branch after every "abf get" (or use -b rosa2012.1). But you can just add an alias like "g: get -b rosa2012.1" and after that "abf g" will do it automatically.

Moreover, the alias name can be not only the first abf argument. For example, you can create another useful alias like "pack: rosa2012.1 build_branch -p" and run "abf copy pack" to copy contents of rosa2012.1 branch to build_branch and compress it.

Options:

abf alias <action> param1 [param2] [...]

  • <action>: One of actions: list, add, remove.

list

List all the currently available aliases. By default this list looks like 

        b: build
       sp: search projects
       su: search users
       st: status
        s: store
      spl: search platforms
       sg: search groups

add

Add a new alias. the first argument is an alias name, all the arguments after that are alias target. For example, abf alias add sg search groups will срфтпу every call of "abf sg rosa" to "abf search groups rosa".

remove

Remove an alias. The only option is an alias name.

clean

Scan the spec and .abf.yml files for possible problems. First, it detects missing sources or patches (which are mentioned in spec using non-remote path, but are absent in current directory and .abf.yml). The check will also fail if there are some problems with the spec file itself and RPM can't parse it. These checks are automatically executed when creating a build task from the current directory. If they fail, the console client refuses to create a build task. This behavior can be turned off by --skip-spec-check option for the build command.

In addition, the 'clean' check reports warnings about files specified in .abf.yml and at the same time in the spec file with remote URL. These warnings are not considered to be crucial and don't prevent the build from being started

get

Clone a remote git repository by its group and name. For example, abf get import/gcc will clone a project gcc from group import.

Options:

abf get <project> [--branch <branch_name>]

  • <project>: a project to clone.
  • -b/--branch <branch_name>: specify a branch to check out inside a cloned git pepository.

Notes:

You can omit a group name, your default group (given from configs) will be used.

put

Upload binary files to File-Store and update (or create) .abf.yml file. Can also commit and push changes.

Options:

abf put [--message <message>] [--minimal-file-size <size>] [--do-not-remove-files]

  • -m/--message <message>: With this option specified, "git add --all", "git commit -m <message>" and "git push" will be executed.
  • -s/--minimal-file-size <size>: The minimal file size to upload to File-Store. Smaller files will not be processed. Default is 0B.
  • -n/--do-not-remove-files: By default files are being removed on uploading. Override this behavior.

store

Upload a given file to File-Store. Prints a sha1 hash or error message (with non-zero return code).

If the file with same hash already presents on File-Store, it will not be reuploaded. In any case, file hash will be printed.

Options: abf store <path>

  • <path>: A path to file to upload.


fetch

Download all the files listed in .abf.yml from File-Store to local directory.

Options:

abf fetch [--only <file_name>]

  • -o/--only <file_name>: Limit the list of downloaded files to this file name(s). This option can be specified more than once.

show

Show some general information about the project. Bash autocomplete uses it. If you are in a git repository directory, command will show information about this project (unless -p/--project specified).

Options:

abf show <target> [--project <project_name>]

  • <target>: What has to be displayed. Available choices: build-repos, build-platforms, save-to-repos, save-to-platforms
  • -p/--project <project_name>: Project to show information about. If you are in a git repository, the information for <project_name> will be displayed instead.

Notes:

Repository name includes its platform name. Example of repository name: "rosa2012.1/main" - repository "main" of "rosa2012.1" platform.

Build repository - repository that you can connect to a building chroot.

Target (or save-to) repository - when a build task completed successfully, packages can be published to this repository.

locate

Command to maintain a database of local projects.

Console client tends to maintain a database of local projects: while any interaction with your local git repository via abf-console-client, it stores the path to this repository. After that you can call abfcd [<project_group>/]<project_name> (omit group_name to use your default) to change your current working directory to the project's path (if it had been stored before). If you just want to get a project path without changing directory, use abf locate -p <project>.

You can also add projects to this database by manually. To add only one project call abf locate update -d /path/to/project. To scan directory recursively and add all the found projects, call abf locale update-recursive -d /path/to/directory.


Options:

abf locate [<action>] [--project <project>] [--directory <directory>]

  • <action>: one of {update, update-recursive}. update will update
  • -p/--project <project>: Project to show information for (if needed). Format: "[group/]name". If no group specified, your default group will be used.
  • -d/--directory <directory>: Directory to update locations for. It should be a git repository for "update" and any directory for "update-recursive". If not specified - the current directory will be used.


build

Initiate a build task on ABF. There are lots of options, but in most cases it works great with no options at all. abf-console-client tries to automatically resolve all the options needed. How does it work: read notes and examples.


Options: abf build [--project <project>] [--branch <branch>] [--tag <tag>] [--commit <commit>] [--save-to-repository <repository>] [--repository <repository>] [--arch <arch>] [--auto-publish] [--update-type <type>] [--skip-spec-check]

  • -p/--project <project>: project name ([group/]project). If the option is not specified and you are in a git repository directory - a project name will be resolved automatically.
  • -b/--branch <branch>: git branch name to build. Read notes.
  • -t/--tag <tag>: git tag to build. Read notes.
  • -c/--commit <commit>: git commit sha hash to build. Read notes.
  • -s/--save-to-repository <repository>: repository to save results to. If no platform part specified, it is assumed to be "<default_group>_personal". If this option is not specified at all and you are in a git repository, it will be resolved automatically (read notes).
  • -r/--repository <repository>: repositories to build with. Can be set more than once. If no platform part specified, it is assumed to be your "<default_build_platform>". If no repositories were specified at all, it will be resolved automatically (read notes).
  • -a/--arch <arch>: architectures to build for. Can be set more than once. If not set - use all the available architectures for the project given.
  • --auto-publish: Enable automatic publishing.
  • --update-type <type>: Update type, one of {security, bugfix, enhancement, recommended, newpackage}. Default is "security".
  • --skip-spec-check: Do not check spec file.

Notes:

Console client tries to automatically resolve all the options without taking into account anything except --branch. If it fails - use only user-given options. If succeeds, but user specified other parameters - discard everything we've resolved and use only user-given options.

Git hash resolving:

  • Only one of --branch, --tag or --commit options can be used at once.
  • If you've specified commit hash - it will be used "as is".
  • If you've specified branch or tag name - commit hash will be resolved automatically using ABF API. (the hash of top commit in the branch will be used for branch)
  • If you've specified a project name - you have to specify a branch, tag or commit manually.
  • If you've specified no branch, tag or commit hash and you've not specified a project name (you have to be in a git repository) - the top remote commit of your current branch will be used.

Other options resolving:

  • if no --arch specified, use all the available project architectures.
  • build repository can usually be resolved from a git branch name. If there is a build platform with the name of your current git branch exists - it will be used.
  • save-to repository: if some repository from a build platform can be used as save-to repository - it will be used.

You can execute abf build (with no options) when you've specified a branch name (or it's resolved automatically) and there is a platform on ABF having the same name. In this case a save-to repository will be selected from a list of available save-to repositories for this platform. Build repositories are repositories of this platform too("main" and one another repository if needed for selected save-to repository).

Examples:

  • Build a current branch of a local cloned project. Build and save-to repositories can be resolved from branch name: abf build
  • Build a project without a local git repository. Build and save-to repositories can be resolved from branch name: abf build --project import/gcc --branch rosa2012.1
  • Build a project to another platform: abf build --project import/gcc --branch rosa2012.1 --save-to-repository rosa2012lts/contrib. Build repositories will be rosa2012lts/main and rosa2012lts/contrib.


mock-urpm

Build a project locally using mock-urpm. No checkouts will be made, the current git repository state will be used.

Options:

abf mock-urpm [-c <config>]

  • -c/--config <config>: A config template to use. Specify one of the config names from /etc/abf/mock-urpm/configs/. Directory path and extension (".cfg") should be omitted. If no config specified, "default.cfg" will be used. Autocompletion worsk for config names.

Notes:

  • Be careful, "abf fetch" will be called before building a package. It means that if you have modified tarball, but .abf.yml contains a hash of an old one - your tarball will be replaced by downloaded from server.
  • Building root can be found in /var/lib/abf/mock-urpm.

rpmbuild

Build a project locally using rpmbuild. No checkouts will be made, the current git repository state will be used.

Options:

abf mock-urpm [-c <config>]

  • -c/--config <config>: A config template to use. Specify one of the config names from /etc/abf/mock-urpm/configs/. Directory path and extension (".cfg") should be omitted. If no config specified, "default.cfg" will be used. Autocompletion worsk for config names.

Notes:

  • Be careful, "abf fetch" will be called before building a package. It means that if you have modified tarball, but .abf.yml contains a hash of an old one - your tarball will be replaced by downloaded from server.
  • Building root can be found in /var/lib/abf/mock-urpm.

publish

Publish the task that have already been built.

Options:

abf publish <task_id> [<task_id>] [...]

  • <task_id>: ID of a build list to publish.


copy

Copy files from one git branch to another. Be careful: all the files from destination branch will be removed, and replaced by the files from a source branch.

This command can be useful to keep sources of the project in one branch and build package in another. Just execute abf

Options:

  • <src_branch>: A branch to copy files from.
  • <dst_branch): A branch to copy files to. If not specified, it's assumed to be the current branch.
  • -p/--pack: Create a tar.gz from the <src_branch> and put this archive and spec file to <dst_branch>.


status

Show information about a build list. It will print something like that: 

 Buildlist ID:       944492
 User:               akirilenko
 Project:            import/mock-urpm
 Status:             build has been published
 Build for platform: rosa2012.1
 Save to repository: rosa2012.1/main
 Build repositories: [rosa2012.1/main]
 Architecture:       i586
 Created at:         2013-02-12 15:25:09
 Updated at:         2013-02-12 15:43:16


Options:

abf status [--project <project>] [--short]

  • -p/--project <project>: Project. If last IDs for this project can be found in local database - use them. Otherwise, print error message end exit.
  • -s, --short: Show only one-line information including id, project, arch and status.


clean

Analyze spec file and show missing and unnecessary files from the current git repository directory.

Options:

abf clean [--auto-remove]

  • -a/--auto-remove: automatically remove all the unnecessary files.

search

Search for something on ABF.

Options: abf search <target> "<query>"

  • <target>: what to search for: {users, groups, platforms, projects}.
  • <query>: A string to search for. No regular expressions, just a plain text.

test

Execute a set of internal datamodel tests.

Additional features

Cache of local project locations

ABF client remembers locations of all repositories ever cloned using it and you can easily move to particular project directory in your local filesystem using "abfcd" command: 

abfcd import/abf-console-client

This information is stored in .abf_projects file in your home directory.

You can learn the location of any project in your filesystem by executing 

abf locate -p PROJECT

If you already have a set of repositories not known to the console client (e,g,m they were cloned using git directly or you removed your .abf_projects file), you can ask the console client to scan particular folders and add all project it finds there to its local cache: 

abf locate update-recursive -d PATH

where PATH is a directory with repositories.

Note also that information from .abf_projects file is used by the console client to move files among different projects.

Bash autocompletion

ABF console client support rich autocompletion in Bash. In particular, it can suggest completions for option names, git branches, build- and save-to repositories for "abf build" and a lot of other parameters. As a result, it's much easier to work with console client, because you don't have to type the long repository name or remember the list of possible variants. In some cases autocompletion may take about 1 second, but remember that its results are always cached, so the next time it will be much more faster.

Retrieved from "http://wiki.rosalab.com/en/index.php?title=ABF_Console_Client&oldid=8228"