Блог:ROSA Planet
A technical blog of ROSA Laboratory.
All content is published under Creative Commons Attribution-ShareAlike 3.0 License (CC-BY-SA)
Please, subscribe to RSS/Atom feed. If you have any questions do not hesitate to contact us
Hardware trends
Today I'm glad to open the next major update of the hardware database — a live statistical report on Linux-powered hardware configurations of our users: https://linux-hardware.org/?view=trends
The report helps to answer questions like "How popular are 32-bit systems?", "How fast is SSD market share growing?", "Which hard drives are less reliable?", "How many computers use old CPU microcode?", "How good is device drivers support?", etc.
In addition to ROSA distribution, other Linux distributions also participated in the study. Most active participants currently are Ubuntu, Mint, Endless, Fedora, Arch, Manjaro, Debian, Zorin, openSUSE, KDE neon, Clear Linux and Gentoo. For top distributions in this list one can find most accurate results.
All charts and table rows are clickable — you can see the details of particular computers counted in statistics. I.e. in addition to statistics and forecasting, the report can be used as a powerful search engine.
The static version of the report for the current month is also available in the Github repository.
The report is built on the basis of user probes with the help of hw-probe (for other distributions: AppImage, Snap, Flatpak and Docker):
hw-probe -all -upload
Please participate!
Probes of the current month are accumulated and appear in the statistics on the first day of the next month. Please let us know if you have ideas for new statistical reports that are not yet implemented in the study.
Search for drivers
There are often cases when a couple of devices does not work properly in your computer out of the box under Linux. The reason for this may be too new hardware (not yet implemented in the kernel), the absence of necessary Linux drivers (not provided by hardware vendors), too obsolete hardware, incompatible devices (e.g. storage controller and drive model, etc.) or a defect. According to data from the Linux-Hardware.org, at least 10% of Linux users encounter such problems. According to our statistics, the most problematic devices are:
- WiFi cards
- Bluetooth cards
- Card readers
- Fingerprint readers
- Smart card readers
- Printers
- Scanners
- Modems
- Graphics cards
- Webcams
- DVB cards
- Multimedia controllers
If a device does not work, then this does not mean that it cannot be configured properly. Sometimes you can find a more suitable kernel or a third-party driver for a device on the Internet. To search for the required kernel, you can use the LKDDb database, where all Linux kernel versions are indexed for supported device drivers, or search for a solution on forums and similar resources.
Today we are launching a new way to find drivers — by creating of hardware probes! If a driver was not loaded for some device in your computer probe, then the database engine will offer a suitable kernel version or known third-party drivers. The same information is presented on each PCI/USB device page in the database.
A probe can be created by the following command (there is an AppImage for other Linux distributions):
hw-probe -all -upload
The drivers search is carried out on the basis of LKDDb database for kernels from 2.6.24 to the newest 5.0 version. Also we have indexed the following third-party drivers:
- nvidia — NVIDIA graphics
- wl — Broadcom WiFi-cards
- fglrx — AMD/ATI graphics
- hsfmodem — modems
- sane — scanners
- foomatic — printers
- gutenprint — printers
- and about 100 drivers from Github for WiFi cards and other devices
Yes, we have indexed fglrx for old AMD/ATI graphics cards. In some cases, the performance of this driver was higher than the free radeon driver, but you need to install a previous version of your Linux distribution, because the fglrx driver is not supported by newest Linux kernels and Linux distributions.
This is not the only example when you need a rollback to an old kernel version. For some devices, the drivers become obsolete and are removed from the kernel. In such cases, you need to pick up one of the old versions of your Linux distribution with an appropriate old kernel, and then manually update necessary software if not provided in the repositories (e.g. browser).
Review of hardware probes
Did you manage to configure a hardware device that did not work out of the box? Did you find the right driver? The device does not work and you don't know what to do? Write a note about your experience right now in your hardware probe!
Registration is not needed — authorization of your computer is done while creating a probe. Just create a probe and immediately open it in your browser. You'll see a big green REVIEW button on the probe page for creating a review.
In the review, you can adjust the automatically detected operability status of a hardware device and write a comment for any device in the Markdown language.
Statuses and comments are assigned to the corresponding devices in the open database. Other users will be able to quickly learn from your experience if they have the same problem by creating probes of their computers and following the links to the database on the probe page.
One can create a probe by the command:
hw-probe -all -upload
Or by the Hardware Probe application in the SimpleWelcome start menu of the ROSA Fresh Linux distribution (KDE4 and Plasma editions). There is also portable AppImage, Docker, Snap and Flatpak to use on any other Linux distribution.
Hardware database for all Linux distributions
The Linux-Hardware.org database has been divided recently into a set of databases, one per each Linux distro. You can choose your favorite Linux distribution on the front page and hide probes and information collected from other Linux distributions.
Anyone can contribute to the database with the help of the hw-probe command:
hw-probe -all -upload
Hardware failures are highlighted in the collected logs (important SMART attributes, errors in dmesg and xorg.log, etc.). Also it's handy to search for particular hardware configurations in the community and review errors in logs to check operability of devices on board (for some devices this is done automatically by hw-probe — see statuses of devices in your probe).
Hardware stats and raw data are dumped to several Github repositories: https://github.com/linuxhw
Thanks to all for attention and new computer probes!
Checking devices operability
We've implemented automated operability checks for devices via analysis of collected system logs in probes. We check if the driver is loaded and used for each device in the probe and if the device performs basic functions. For network cards we check received packets, for graphics cards we check absence of critical errors in the Xorg log and dmesg, for drives we check S.M.A.R.T. test results, for monitors we check the EDID and for batteries we check the remaining capacity.
The operability status is detected for the following devices:
- Graphics cards
- WiFi cards
- Ethernet cards
- Bluetooth cards
- Modems
- Hard drives
- Monitors
- Batteries
- Smart card controllers
For the following devices we can only detect if the device is failed to operate:
- Sound cards
- Card readers
- Fingerprint readers
- TV cards
- DVB cards
You can check all your previous probes now — the statuses are already updated!
If you are a new user, then you can create a probe by the hw-probe command:
hw-probe -all -upload
Nonworking devices are collected in the Github repository: https://github.com/linuxhw/HWInfo
Thanks to all for attention and new computer probes!
EDID repository
The largest open repository of monitor characteristics has been created recently containing EDID structures for more than 9000 monitors: https://github.com/linuxhw/EDID
EDID (Extended Display Identification Data) is a metadata format for display devices to describe their capabilities to a video source. The data format is defined by a standard published by VESA. EDID data structure includes manufacturer name and serial number, product type, phosphor or filter type, timings supported by the display, display size, luminance data and (for digital displays only) pixel mapping data.
The most famous analogue of the repository is the EDID.tv project, which also contains quite a lot of information about monitors.
The repository is replenished automatically based on recent hardware probes. One can participate in the replenishment of the repository by executing of one simple command in the terminal:
hw-probe -all -upload
The hw-probe utility is pre-installed in the ROSA Linux distribution. Users of other systems may use AppImage, Docker-image, LiveCD or other techniques to create probes.
HW Probe 1.4
Friends, I'd like to introduce new hw-probe 1.4.
Most significant change in this release is the anonymization of probes on the client-side. Previously "private data" (like IPs, MACs, serials, hostname, username, etc.) was removed on the server-side. But now you do not have to worry how server will handle your "private data", since it's not uploaded at all. You can now upload probes from any computers and servers w/o the risk of security leak.
The update is available in repositories.
Other changes:
- Up to 3 times faster probing of hardware
- Collect SMART info from drives connected by USB
- Initial support for probing drives in MegaRAID
- Improved detection of LCD monitors and drives
- Collect info about MMC controllers
- Probe for mcelog and cpuid
- Etc.
You can, as before, create a probe of your computer via the application in SimpleWelcome menu or from the console by a simple command:
hw-probe -all -upload
Thanks to all for attention and new probes of computers!
Highlighting important SMART attributes in probes
We've started to highlight most important SMART attributes in computer probes, that correlate with real mechanical failures according to Google and Backblaze studies.
Green highlights the zero value of important attributes, red — any positive value. You can review your probes now!
You can find links to the studies, as well as a complete list of highlighted important attributes here: https://github.com/linuxhw/SMART
You can, as before, create a probe of your computer via the application in SimpleWelcome menu or from the console by a simple command:
hw-probe -all -upload
Thank you all for the attention and new hardware probes!
List of devices with poor Linux-compatibility
A new project has been created to collect the list of computer hardware devices with poor Linux compatibility based on the Linux-Hardware.org data for 4 years: https://github.com/linuxhw/HWInfo
There are about 26 thousands of depersonalized hwinfo reports in the repository from computers in various configurations (different kernels, OS — mostly ROSA Fresh). The device is included into the list of poorly supported devices if there is at least one user probe in which the driver for this device was not found. The column 'Missed' indicates the percentage of such probes. If number of such probes is small, it means that the driver was already added in newer versions of the OS. In this case we show minimal version of the Linux kernel in which the driver was present.
Devices are divided into categories. For each category we calculate the ratio of poorly supported devices to the total number of devices tested in this category.
At the moment, the study is limited only to PCI and USB devices. In the future, it is planned to include the rest.
Please check the presence of known unsupported devices in the table. The device ID can be taken from the output of the 'lspci -vvnn' command in square brackets, for example [1002:9851].
Real-life reliability test for hard drives
A new open project has been created to estimate reliability of hard drives (HDD/SSD) in real-life conditions based on the SMART data collected in the Linux-Hardware.org database. The initial data (SMART reports), analysis methods and results are publicly shared in a new github repository: https://github.com/linuxhw/SMART. Everyone can contribute to the report by uploading probes of their computers by the hw-probe tool!
The primary aim of the project is to find drives with longest "power on hours" and minimal number of errors. We use the following formula as a measure of reliability: Power_On_Hours / (1 + Number_Of_Errors), i.e. time to the first error/between errors.
Please be careful when reading the results table. Pay attention not only to the rating, but also to the number of checked model samples. If rating is low, then look at the number of power-on days and number of errors occurred. New drive models will appear at the end of the rating and will move to the top in the case of long error-free operation.
You can, as before, create a probe of your computer via the application in SimpleWelcome menu or from the console by a simple command:
hw-probe -all -upload
Docker Image for ROSA Desktop Fresh R9
We are opften asked where to get Docker image for ROSA Desktop Fresh. For example, such an image would be useful if you want to quickly debug build of some package and don't have a machine with ROSA installed near you. It doen't look sane to setup a Virtual Machine for a one-time task, while Docker container perfectly fits most of such needs.
For rosa2016.1 platform, Docker images are available here:
https://hub.docker.com/r/dsilakov/rosa/ https://hub.docker.com/r/sibsau/rosa/
If you need 2014.1 one, you can find it there:
https://hub.docker.com/r/fgagarin/rosa/
Feel free to launch docker container in an ordinary way:
- docker run -it --rm dsilakov/rosa
Files and scripts that can be used to create such images can be found in ABF: https://abf.io/soft/docker-brew-rosa
Pull requests and suggestions are welcome!
ROSA Image Writer
Optical drives are rapidly disappearing from our computers of all kinds, and consequently installing operating systems from USB flash disks is becoming increasingly popular. ISO images of the ROSA distribution were originally intended for burning to DVD disks, but they can as well be written to flash disks which would allow you to boot from them and launch the Live system or start installation. There is no standard tool for writing images to flash disks, everybody uses what he or she likes. In ROSA the command line tool dd was traditionally recommended for performing this kind of job. However, it can hardly be called user-friendly, and most users would feel at least some discomfort, if not terror, using it. For Windows users the situation is even worse. Granted, there is a dd port for Windows, but it happened to have serious bugs which prevent the resultant flash disks from working properly. All this led to the solution of developing our own tool, ROSA Image Writer.
The first version was based on the Windows version of SUSE Studio Image Writer, but its C# language (and, naturally, requirement for .NET Framework), usage of two completely unrelated projects in two different frameworks for Windows and Linux versions, and some other drawbacks made us write the new tool from scratch. Now ROSA Image Writer is developed in C++ with Qt5 framework and supports both Windows and Linux from the same codebase.
The list of main functions is:
- Selecting the image file via either usual Open File dialog, or by drag&dropping the file on the application window.
- The list of USB devices shows for each device its user-friendly name, size, and logical disks that originate from the device.
- When user inserts or removes a USB device, the list is refreshed automatically.
- When writing is in progress, the progressbar is displayed which is also translated onto the taskbar button in Windows 7/8.
- The application supports localization and includes the Russian translation.
Download links are available on the description page:
Helping ROSA Maintainers To Update Programs
One of the most frequent kind of ROSA users' requests is request to update some package in our repositories. Unfortunately, we can't satisfy all update requests immediately. However, every community member can easily help us with it, since with our current build infrastructure you can update packages even if you are not programmer or maintainer at all. In many cases basic knowledge of web browser is enough.
Assume that you have noticed a new release of your favorite application and want to update corresponding package in ROSA repositories. Consider for example a popular Atom code editor - right now ROSA repositories provide atom 1.2.0, but 1.3.2 was released several days ago. Let's try to update the ROSA package!
First of all, please register at our ABF build environment at http://abf.io. Registration is free, doesn't require any kind of confirmation and you will be able to log in into ABF just after registration is complete.
Then please find a project corresponding to your package - in our case we need atom project in the import group (always use the import group since it is contains projects built into our official repositories!).
Go to the project found and press "Fork".
In the dialog appeared choose Fork to <your_login>/atom. This will fork the project into your personal namespace where you are free to modify it as you wish. The forking is fast and you will be immediately redirected to a page of the forked project. If you see a message that repository is empty then just reload the page.
NExt step is to switch to the rosa2014.1 branch of the Git repository. If you doesn't understand what this means - don't worry, you just have to choose "rosa2014.1" in dropdown list at the upper-right corner of the page. We use "rosa2014.1" branch starting from ROSA Desktop Fresh R4 and will continue to use it at least in R7.
Now please find a file with the .spec extension among project files (atom.spec in our case), click on it and then click Edit to modify it.
All that you should do in this file is to update Version tag which is usually located near the top of the file. As we can see, currently this tag is set 1.2.0 and we will replace this value with 1.3.2. Please also write some valuable message (e.g., "Updated to version 1.3.2") in the "Commit message" text area. And that's all - save file by means of corresponding button at the bottom of the page.
In the previous step we have prepared ABF to build a new version of our program. Now it is time to click on "New Build" and perform a couple of actions in the page appeared:
- detach your personal repository from the build by clicking on a cross sign near it
- attach Contrib repository by activating "contrib" checkbox item in the "rosa2014.1" section
Now press "Start Build" and wait for result. If the build fails - well, the easy-way update didn't work in this case and it is time to ask maintainers or improve your own skills and dig deeper into package build procedure. But if the build succeeds, you will be able to download packages with the new version of your application. Feel free to download and test them.
If the new version works fine, feel free to share this updates with the whole ROSA community. To do this, just press the "Pull Request" button at your project page.
Make sure that you have specified "rosa2014.1" branch for both source and target projects, press "Update commits" button if it becomes visible, write some message in the "Description" field and click on "Send Pull Request".
ROSA maintainers will receive you request, review it and likely accept and build the updated package into official repositories.
