When told to force load a module, we were removing only the value of
vermagic instead of the complete entry.
Philippe De Swert (philippe.deswert@jollamobile.com) sent a patch that
was additionally mangling also the last two chars of the key
("vermagic="). Instead of creating an invalid entry in .modinfo section
like this, this patch removes the complete entry, key + value, by
zeroing the entire string.
Much thanks to Philippe who found the issue and pointed to the fix.
If we are accessing several times the modules and reading some sections
by sucessive calls to the functions below, we are incurring in a penalty
of having to open, parse the header and close the file. For each
function.
- kmod_module_get_info()
- kmod_module_get_versions()
- kmod_module_get_symbols()
- kmod_module_get_dependency_symbols()
These functions are particularly important to depmod. It calls all of
them, for each module. Moreover there's a huge bottleneck in the open
operation if we are using compression. Every time we open the module we
need to uncompress the file and after getting the information we need we
discard the result. This is clearly shown by profiling depmod with perf
(record + report), using compressed modules:
64.07% depmod libz.so.1.2.7 [.] 0x00000000000074b8 ◆
18.18% depmod libz.so.1.2.7 [.] crc32 ▒
2.42% depmod libz.so.1.2.7 [.] inflate ▒
1.17% depmod libc-2.16.so [.] __memcpy_ssse3_back ▒
0.96% depmod [kernel.kallsyms] [k] copy_user_generic_string ▒
0.89% depmod libc-2.16.so [.] __strcmp_sse42 ▒
0.82% depmod [kernel.kallsyms] [k] hrtimer_interrupt ▒
0.77% depmod libc-2.16.so [.] _int_malloc ▒
0.44% depmod kmod-nolib [.] kmod_elf_get_strings ▒
0.41% depmod kmod-nolib [.] kmod_elf_get_dependency_symbols ▒
0.37% depmod kmod-nolib [.] kmod_elf_get_section ▒
0.36% depmod kmod-nolib [.] kmod_elf_get_symbols
...
Average of running depmod 5 times, dropping caches between them, in a
slow spinning disk:
Before: 12.25 +- 0.20
After: 8.20 +- 0.21
m-i-t: 9.62 +- 0.27
So this patch leads to an improvement of ~33% over unpatched version,
ending up with 15% speedup over module-init-tools.
Although the hash table implementation allows passing a callback function
to free a value when it is removed from the hash table, hash_del() wasn't
freeing it if it was provided. Now it does.
As a bonus, it now checks if the callback is set in hash_add() as well.
This is a broken option that only leads to misery and incompatabilities
with other systems. Kbuild doesn't come close to supporting directories
other than /lib/modules with several targets simply failing without
hacky fixes. Simply remove the option and all traces of it, as it
doesn't make sense in today's world.
With this flag kmod_module_probe_insert_module() check if module is
blacklisted only if it's also an alias. This is needed in order to allow
blacklisting a module by name and effectively blacklisting all its
aliases as module-init-tools was doing.
Before this patch we could load pcspkr module as follows:
/etc/modprobe.d/test.conf:
alias yay pcspkr
blacklist pcspkr
$ modprobe yay
Now libkmod has support to blacklist "yay" because "pcspkr" is blacklisted.
Only the public header maintains #ifndef in the header, together with
pragma. The other ones contain only pragma.
As reported by Shawn Landden on systemd mailing list this is compatible
with all major compilers and gcc has this since version 3.3.
It makes more sense to have libkmod-config.c deal with the configuration
directly and the others get the config from ctx. As a bonus point we get
a smaller binary. Following numbers are for x86-64, libkmod + kmod:
Before:
text data bss dec hex filename
128840 1496 104 130440 1fd88 tools/modprobe
After:
text data bss dec hex filename
128392 1496 104 129992 1fbc8 tools/modprobe
I hate this kind of READV and WRITEV macros that Gustavo seems to love.
clang-analyzer hates them as well.
I'm not motivated enough to refactor this, but I want a clean clang
report, so just shut it up.
If we don't have --gc-sections support, linking kmod fails:
libkmod/.libs/libkmod-util.a(libkmod-util.o): In function 'underscores':
libkmod/libkmod-util.c:117: undefined reference to 'kmod_log'
This is because libkmod-util.la uses kmod_log(), that is in libkmod.la.
Move the function so we don't have a dependency loop while building the
libraries and it works with compilers with no support for --gc-sections.
There's no reason kmod_log should be exported, remove it from linker
script. This doesn't break the API/ABI because we are luck: since the
function had visibility=hidden it was not getting exported as a global
symbol.
This reverts commit 88a170dbd6.
There's no reason for users of the API to call this method, it's just
wrong to export it.
The bug that this patch fixed needs to be fixed another way, not
exporting this function.
zlib won't necessarily set the system errno, and this is particularly
evident on corrupted data (which results in a double free). Use zlib's
gzerror to detect the failure, returning a generic EINVAL when zlib
doesn't provide us with an errno.
If we don't have --gc-sections support, linking kmod fails:
libkmod/.libs/libkmod-util.a(libkmod-util.o): In function 'underscores':
libkmod/libkmod-util.c:117: undefined reference to 'kmod_log'
This is because kmod_log is missing the export define, even though it's
already listed in the exported symbol list.
This matches the change in systemd and udev. Log message on udev's
change by Kay Sievers:
After long consideration we came to the conclusion that user
configuration in /etc should always override the (generally
computer generated) configuration in /run. User configuration
should always be what matters over anything else. Hence rearrange
the search orders accordingly. In general this should change
very little as overriding like this is seldomn done so far,
and the order between /etc and /usr stays the same.
If kernel doesn't have support to unload modules,
/sys/module/<modname>/refcnt will not exist and that's ok.
Reported by: Sven Anders <anders@anduras.de>
This is a more generic method of applying filters to module lists. This
deprecates kmod_module_get_filtered_blacklist() which now simply returns
a call to _apply_filter with the extra filter enum arg.
Running two instances of modprobe with the same module should both
succeed or both fail:
modprobe foo&; modprobe foo;
Previously if foo failed to be inserted by the first call, the second one
could return 0 because it may have occurred while the first one was being
processed by kernel (thus marked as "coming").
Now we simply don't check by "coming" in order to decide if we need to
call init_module(). module-init-tools used to spin calling
usleep(100000), but calls to init_module() are already synchronous.
Therefore let kernel synchronize the calls.
Search modules.builtin file before saying the module was not found.
Note: these "modules" should not appear as dependencies of other modules
(in modules.dep) even if they appear in modinfo. This fixes the return
code of modprobe with builtin modules.
Also fixes a small coding style issue in module_is_inkernel().
If a softdep depends on a module in the dependency list of the module
being inserted, we would enter and infinite loop.
Move the "mod->visited = true" assignment to the proper place, hoping it
didn't break other use cases. This is a bug that comes and goes every
now and then. Since we have a testsuite now, a test for this should be
written.
Commit "af9572c lib/module: check initstate before inserting module"
removed the check for "we should return -EEXIST" and moved it back to
the start of the function. The problem with this is the following
scenario:
- We check if module is in kernel -> no
- We insert the dependencies
<-- External program loads
the module
- We check if module is in kernel -> yes
- We return 0, when we should return -EEXIST
Use a function to properly get an unsigned short from memory that is
possibly unaligned.
Note that it implicitly fixes a small bug in the hash function that
was introduced when modifying the eina code: the line "hash ^= key[2]
<< 18;" is supposed to be accessing the 3rd byte of the remainder of
the input, but when 'it' was introduced, 'key' ('data' in eina code)
was no longer incremented, so this ended up accessing the 3rd byte of
the input from the beginning. This is fixed by iterating over 'key',
like the eina code does.
Before this patch depmod was failing on ARMv5 and possibly others that
don't have unaligned access. They do not calculate correctly the
dependencies as shown below:
[root@alarm ~]# modinfo bridge
filename: /lib/modules/2.6.39.4/kernel/net/bridge/bridge.ko
version: 2.3
license: GPL
srcversion: 6B583530AE2B39C7E2317BF
depends: stp,llc
vermagic: 2.6.39.4 preempt mod_unload ARMv5
[root@alarm ~]# depmod
[root@alarm ~]# cat /lib/modules/2.6.39.4/modules.dep |grep bridge
kernel/net/bridge/bridge.ko:
[root@alarm ~]#
See how modinfo properly lists the dependencies, but modules.dep which
depmod generates does not contain them. As a result, most kernel
modules fail to load because their dependencies are not loaded by
modprobe.
This applies to both the high level probe_insert_module() and the
underlying insert_module() functions. By checking module initstate prior
to inserting a module, we can avoid a lot of needless work just to find
out that the init_module call fails with EEXIST.
This implements a helper function, module_is_inkernel, to return a
boolean value describing if a module is live, coming, or builtin.
Just printing the errno string such as "%m\n" is not enough to help
debug or users understand the problem.
Change to provide more context on the failing operation.
Some messages may happen more than once in the same function and
discovering the line is hard. Now we print the actual log priority
that exposed the message as well as filename and line.
NOTE: We should consider printing the log priority in the non-debug
version as well.
We need a way to tell libkmod to ignore loaded modules, so modprobe can
tell it to dry-run and show dependencies. However there's a conflict
with two flags. KMOD_PROBE_STOP_ON_ALREADY_LOADED prevails if passed
together with KMOD_PROBE_IGNORE_LOADED.
It's not as simple as tell user to check if the module is loaded before
calling this function. Due to race conditions, module might not be
loaded before the function call, but fail later because another process
inserted it.
Share code of module creation among the several new functions. With this
we let the alias/modname/path parsing to the separate functions, and the
rest with the common one.
This fixes the issue of alias names not being able to contain dots.
Alias names may contain dots. However since kmod_module_from_alias()
still calls kmod_module_new_from_name(), the bug is not entirely fixed,
and will be completely corrected in a later patch.
Split kmod_module_probe_insert_module() in 2:
1) Get list of modules to be loaded
2) Iterate the list, loading the module
With this in future we will be able to cover use cases of modprobe,
that has a logic a bit more complicated.
With this we also change the logic to detect dependency loops: instead
of checking the recursion every STEP times, we now keep a field in
kmod_module, marking it as visited. We simply ignore already visited
modules and thus we break loops.
This field can be used to iterate the modules, controlling whether we
are revisiting a certain module. A function to clear the values in all
modules is needed since when we are iterating, we don't know if the
module is created anew or if it's picked from the pool. Therefore we
can't know if the field is true because of a previous iteration or if
the module was indeed already visited.
Not all libc's have a mtim member in struct stat (dietlibc doesn't).
Change ts_usec() to receive a struct stat as parameter and implement it
accordingly for both cases.
Index dump doesn't use stdio.h function and instead call write()
directly on STDOUT_FILENO file descriptor. Therefore we need to flush
stdio buffers before calling it, to be sure the configuration dump will
appear before index's.
Provide a function to dump the index files to a certain fd. It could be
more optimized (particularly the functions to dump the index that were
copied and pasted from m-i-t), but it seems like the only user of it is
'modprobe -c', used for debugging purposes. So, keep it as is.
Config iterators are useful to get each configuration list, remember its
type and how to get their key/value pair.
softdeps don't have the value yet, because they are stored as string
vectors.
Current limitation is horrible no support to sections: we have to to
have separate header files or to maintain the libkmod-sections.txt file.
We are doing the latter.
Module aliases can be bigger than NAME_MAX. So, replace with PATH_MAX
that is bigger enough to hold them.
Technically in some places NAME_MAX would be sufficient (those using
module names only), but they use functions that can be called with
alias. So increase the buffers in these cases to PATH_MAX too.
Refactor code to use pointer to functions, avoiding the previous
Now comp_types defines a magic header to be checked (size and bytes),
with the associated load() and unload() operations. If a header
matches, their operations are used. Otherwise the regular file
operations (mmap/munmap) are used.
File descriptor close is managed by the common code if it's valid
(>=0). If some code steals the file descriptor (eg: gzopen), then they
must change file->fd to -1.
This way the code should be easier to extend and avoid bugs.
ouch, I did a mess in the original function, fix them:
* on errors (read() < 0), continue reading after the done bytes, not
at position 0.
* read buflen - 1 bytes, so there is always room to store the
trailing \0, as expected by user due behavior of snprintf(),
fgets() and others.
Return -errno instead of the value returned by init_module(). We need to
differentiate between the several errors that might occur, e.g. "module
already loaded", access denied, etc.
Treat module insertion as modprobe does: look for (soft-)dependencies, run
install commands, apply blacklist.
The difference with the blacklist is that it's applied to all modules,
including the dependencies. If you want to apply a blacklist only on the
module it's better to call the filter function by yourself.
This implementation detects loops caused by poorly written
soft-dependencies and fail gracefully, printing the loop to the log.
It only worked because n was always 1. kmod_list_remove returns a
pointer to the next element, relative to the removed one. Therefore we
need to always get a pointer to the last.
build: explicitly call PKG_PROG_PKG_CONFIG
Per the manual page, PKG_PROG_PKG_CONFIG needs to be invoked
explicitly if PKG_CHECK_MODULES might not happen (it is indeed stowed
in an AS_IF in kmod). Without this, funny failures can occur.
(As it did.)
Uses kmod_elf_get_dependency_symbols() that looks into ".symtab" for
UNDEF symbols and matches the name from ".strtab" to "__versions" to
get crc.
Likely the public API should unify the symbol information getters and
list release, they are almost the same.
Similar to module-init-tools load_symbols(), it will try .symtab and
.strtab for symbols starting with __crc_, if they are found their crc
is read from ELF's Elf_Sym::st_value.
If not found, then it will fallback to __ksymtab_strings.
Just now realized that my distro (Gentoo) enables support for gzip but
does not compress modules by default.
In this case it's better to have a special case that uses mmap()
instead of a loop of realloc() + gzread().
We need to keep config files sorted and use them taking the precedence
order into account.
The following message was taken from module-init-tools commit doing a
similar thing:
Configuration files are parsed in alphabetic order, regardles of
what directory they reside in. Furthermore, if several files by
the same name exist in different directories only the one in the
directory with highest precedence is loaded.
The order of precedence is /run, /etc, /usr/lib, /lib.
The sad thing is that we are not using openat() anymore since each file
is in different directories. In future we might change the
implementation to open all DIRs and keep a reference
to them instead of the path. However we'd have to keep a separate list
with all the opened dirs so we can close them later (when all configs
are parsed).
Check if '=' appeared before the dot. In this case, it's not a valid
module option in kernel command line.
The command line that was failing is:
"root=/dev/sda3 ro pcie_aspm=force init=/sbin/bootchartd
initrd=../initramfs-linux.img BOOT_IMAGE=../vmlinuz-linux"
Commit "b20dc17 Remove unneeded reference to last string" reverted the
fix in "47a0ef6 elf: do not output empty strings." and empty strings are
appearing again in kmod-modinfo.
With this commit we do a bit different and instead of keeping the
reference to last string we skip the '\0' inside the loop.
module-init-tools modprobe.c use fnmatch() and not strcmp() to match
commands and softdeps, although the man page does not say so. Then use
the same function to provide compatibility.
kmod_list_prev() should return NULL if the current element is the
head, not if the previous element is the head. This was likely a copy
& paste error from kmod_list_next().
Install and remove commands are now properly treated on lookup. Example
config file:
$ ./test/test-lookup installme
libkmod version 1
Alias: 'installme'
Modules matching:
installme
install commands: 'echo "this is a install message"'
$ ./test/test-lookup removeme
libkmod version 1
Alias: 'removeme'
Modules matching:
removeme
remove commands: 'echo "this is a remove message"'
modprobe from module-init-tools does not use more than one
install/remove command, it just stops on the first one. Test
modprobe.conf:
install bla echo "this is a message"
install bla echo "this is a message"
$ modprobe bla
this is a message
$
Install and remove commands are already a legacy thing we need to carry,
but let's not extend it so people do not start doing crazy things.
With this patch we are breaking on the first element we find in the
configuration. May be we can add a warning later when parsing the config
that install commands are duplicated.
Module was never being removed from hash table. Therefore, if we create
a module, unref it and create it again we will access freed memory.
Commit "53385cf Improve test of double references" introduced a new test
in test-mod-double-ref.c that previously to this commit was crashing and
now it's working fine.
The hash key is not copied so we can't change the string from:
modname/modalias
to:
modname'\0'modalias
in order to setup mod->name and mod->alias.
Now what we do is:
1) if name is in the form 'modname/modalias', the final struct
kmod_module will be:
struct kmod_module {
char *alias;------,
char *name;-----, |
char *hashkey;--|-|-,
} | | |
name <------------------' | |
alias <-------------------' |
hashkey <-------------------'
2) if name is in the simple form 'modname', then the final struct
kmod_module will be:
struct kmod_module {
char *alias;------> NULL
char *name;-----,
char *hashkey;--|---,
} | |
name <------------------*---'
1 alias may correspond to more than 1 module. This would cause a
conflict in the hash table when inserting a module there and bad things
could happen.
Now we use 'modname/aliasname' as key, '/aliasname' part being optional.
Internally kmod_module_new_from_alias() will setup a 'modname/aliasname'
string and pass to kmod_module_new_from_name() that will treat the case
with a '/' in the name.
User might call kmod_module_new_from_name() without any slashes, so the
key my not contain it.
We have a static vector of indexes. There's no need to check if they
have a suffix or if they are absolute, we just already know in advance
and there are no plans to change it.
Remove function kmod_resolve_alias_options since it's not needed
anymore. Test is using the following configuration file:
alias blablabla ac
options ac test=1
options blablabla test=2
Lookup test by module name:
$ ./test/test-lookup ac
libkmod version 1
Alias: 'ac'
Modules matching:
ac
options: 'test=1'
Lookup test by alias:
$ ./test/test-lookup blablabla
libkmod version 1
Alias: 'blablabla'
Modules matching:
ac
options: 'test=1 test=2'
This function will create a new kmod_module() by calling
kmod_module_new_from_name(), but instead of given the module name, it
gives the alias. This way, the modules' hash will contain the alias
instead of the name. If module was created successfully, then it swap
the alias with the actual name.
The downside is that the structure is not shared anymore if we create a
kmod_module by alias and after by name. However, in modprobe's
configuration it's possible to have different options for different
aliases. In future we might want to create a way to share the common
part of the structure (then having only one instance as we had before).
We still have name allocated just after the struct kmod_module, but
now we use a pointer instead of putting name as a vector in the end of
the structure.
The previous way has some problems, the worst is:
- it's not possible to swap the name with another value: this is
the real problem that this patch is solving. Later patches
will make name be swappable with alias, which is not possible
if name is a vector.
When normalizing alias names (or if we don't know if it's an alias or
modname), use alias_normalize() instead of modname_normalize(). The
difference is that alias names can contain dashes withing brackets, and
those should not be changed to underscores.
Most of the places using underscores() function might be converted to
alias_normalize(), but this is not done now.
It's not possible to move functions related to "live" modules to
libkmod-loaded.c because they depend on the definition of kmod_module.
Putting this structure in the private header is not a good idea, so let
all functions related to "live" information in the end of
libkmod-module.c, and move the sole function from libkmod-loaded.c to
this place. This way all functions get the right documentation
about their sections.
Lines should not go over 80 chars with a few exceptions:
- headers
- function definitions with only 1 argument
- long strings, otherwise we break grep
This should go later in a coding-style file
This will be required to implement modprobe later. The implementation
follows "man modprobe.conf" and allows options to be specified for
alias as well, thus the need for kmod_resolve_alias_options().
Example mod-a.conf:
options mod-a a=1 b=2
options mod-a c=3
alias mymod-a mod-a
options mymod-a d=4
Results in:
options mod-a a=1 b=2 c=3
options mymod-a a=1 b=2 c=3 d=4
Install commands are being concatenated with ";", but manpage is not
clean about this behavior.
For mmap mode, we can avoid allocating and copying strings from the
mmap'ed memory.
With that we have fixed length "struct index_mm_value" that can be
allocated inline with "struct index_mm_node".
uint32_t reads must be aligned, they're not then use memcpy().
read_alloc_chars_mm() and read_chars_mm() were wrong, normalize all
address calculation using single byte pointer "addr" that is
incremented by the amount read, this will avoid further errors.
Lucas De Marchi
Lucas De Marchi
Leandro Pereira
Dave Reisner
Mike Frysinger
Kay Sievers
Dan McGee
Ambroz Bizjak
Gustavo Sverzut Barbieri
Pedro Pedruzzi
Thierry Vignaud
Dave Reisner
Leandro Pereira
Cristian Rodríguez
Jan Engelhardt
Luis Felipe Strano Moraes
Kay Sievers
Ulisses Furquim