Everything

php

Semaphores - feature request

php.internals

Wojciech Malota

19 years ago

It's impossible to correct remove semaphores with sem_remove function
when I use them to provide execution of concurrent processes.
When the last process releases the semaphore I should be able to remove
it. But I don't know if another process haven't acquired the semaphore.
For safety reasons I don't remove released semaphores from system.
Now, If I get 128 semaphores and I execute sem_get one more time I will
get warning.
So, I can't remove semaphores (because I don't know if another process
is using it) and I can't get next semaphore.
I will try to tell about my sugestion.

I assume that I have a function my_sem_remove($sem_id):

function my_sem_remove($id) {
if(Semaphore $id isn't acquired and its FIFO is empty)
sem_remove($id);
}
None of instructions in this function can be interrupted.
So, now the problem is simple to resolve:

$id = sem_get(ID);
sem_acquire($id);

CRITICAL REGION

sem_release($id);
my_sem_remove($id);

But now, two instructions: sem_release and my_sem_remove are not needed.
It's simple to replace them by one function my_sem_release:

function my_sem_release($id) {
sem_release($id);
my_sem_remove($id);
}

For reasons which about I will write later those instructions can't be
interrupted.

So, my example look like this:

$id = sem_get(ID);
sem_acquire($id);

CRITICAL REGION

my_sem_release($id);

But let's imagine two concurrent processes executing code above:

PROC1: $id = sem_get(ID);
<---  interrupt
PROC2: $id = sem_get(ID);
sem_acquire($id);
CRITICAL REGION
my_sem_release($id);
<--- interrupt
PROC1: sem_acquire($id); <--- Uuups!! Error: semaphore doesn't exists,
PROC2 has removed it.

My solution:
Instructions sem_get and sem_acquire can't be interrupted.
So, I assume that there is a function my_sem_acquire:

function my_sem_acquire($key, $max_acquire = 1, $perm = 0666,
$auto_release = true) {

$id = sem_get($key, $max_acquire, $perm, $auto_release);
sem_acquire();
return $id;
}

None of instructions in function above can be interrupted.

And finally correct example:

$id = my_sem_acquire(SEMKEY);

CRITICAL REGION

my_sem_release($id);

It's the best solution, without danger of operating on not existing
semaphores. Also, unused semaphores are automaticcally removed. The
problem is, that functions my_sem_acquire and my_sem_release can't be
witten in PHP so it must be done in C inside source of PHP.

Wojciech Malota

Michael B Allen

19 years ago

On Tue, 2 Jan 2007 08:35:10 +0100
"Wojciech Malota" <wojtekm86@konto.pl> wrote:

> It's impossible to correct remove semaphores with sem_remove function
> when I use them to provide execution of concurrent processes.
> When the last process releases the semaphore I should be able to remove
> it. But I don't know if another process haven't acquired the semaphore.
> For safety reasons I don't remove released semaphores from system.
> Now, If I get 128 semaphores and I execute sem_get one more time I will
> get warning.
> So, I can't remove semaphores (because I don't know if another process
> is using it) and I can't get next semaphore.
> I will try to tell about my sugestion.
>
> I assume that I have a function my_sem_remove($sem_id):
>
> function my_sem_remove($id) {
>   if(Semaphore $id isn't acquired and its FIFO is empty)
>     sem_remove($id);

<snip>

> It's the best solution, without danger of operating on not existing
> semaphores. Also, unused semaphores are automaticcally removed. The
> problem is, that functions my_sem_acquire and my_sem_release can't be
> witten in PHP so it must be done in C inside source of PHP.

I don't understand. If your "FIFO" is some kind of refcount that won't
work because the refcount would not be shared by multiple separate
PHP processes. You would have to put the refcount in shared memory
and even then there's no guarantee that the semaphore will be cleaned
up properly because any exit routine will not run if the application
is killed, crashes, aborts, etc.

The left-over SysV semaphore problem is notoriously impossible. There
basically is no solution other than to simply use something other than
SysV semaphores such as process-shared POSIX semaphores (not as portable
as SysV semphores).

Mike

--
Michael B Allen
PHP Active Directory SSO
http://www.ioplex.com/

Wojciech Malota

19 years ago

> I don't understand. If your "FIFO" is some kind of refcount that won't
> work because the refcount would not be shared by multiple separate
> PHP processes. You would have to put the refcount in shared memory
> and even then there's no guarantee that the semaphore will be cleaned
> up properly because any exit routine will not run if the application
> is killed, crashes, aborts, etc.

FIFO in this context is a queue of processes which are waiting for acquiring
occupied semaphore.
This queue is managed by operating system and is implemented in mechanism of
semaphores.
I think that there must be possibility to check if this queue is empty.

Wojciech Malota

Michael B Allen

19 years ago

On Tue, 2 Jan 2007 20:04:19 +0100
"Wojciech Malota" <wojtekm86@konto.pl> wrote:

> > I don't understand. If your "FIFO" is some kind of refcount that won't
> > work because the refcount would not be shared by multiple separate
> > PHP processes. You would have to put the refcount in shared memory
> > and even then there's no guarantee that the semaphore will be cleaned
> > up properly because any exit routine will not run if the application
> > is killed, crashes, aborts, etc.
>
> FIFO in this context is a queue of processes which are waiting for acquiring
> occupied semaphore.
> This queue is managed by operating system and is implemented in mechanism of
> semaphores.
> I think that there must be possibility to check if this queue is empty.

I'm still very confused. I think your terminology is inaccurate.

If semop is called with sembuf.sem_op < 0 to indicate that the semaphore
should be decremented then a semphore value of 0 will cause the caller to
wait. Is this what you mean by "FIFO" and queue of processes waiting to
"acquire" the semaphore?

If yes, then it would not matter if there were no callers waiting
because that is a valid condition during which the semaphore should not
be removed.

For example, one process may call semop decrement the semaphore to
0 (to mean "locked"), return and then go off to perform work in the
"critical section". At that point there are no callers waiting. Now if
a second process calls semp it will block because the semaphore value
is 0. When the first process completes its work it will call semop with
sembuf.sem_op > 0 to increment the semaphore value. That will signal
the second process to wake up, find the semaphore value greater than 0,
decrement it, and continue on to perform it's critical section.

So the point is that you cannot remove the semaphore when there are no
callers waiting in semop because that is a perfectly valid state.

Mike

--
Michael B Allen
PHP Active Directory SSO
http://www.ioplex.com/

Wojciech Malota

19 years ago

> So the point is that you cannot remove the semaphore when there are no
> callers waiting in semop because that is a perfectly valid state.

Yes, but if function which acquires a semaphore is constructed as
my_sem_acquire function it will be safe to remove the semaphore when it is
released (if semaphore is "unlocked").

Wojciech Malota

Michael B Allen

19 years ago

On Wed, 3 Jan 2007 01:20:41 +0100
"Wojciech Malota" <wojtekm86@konto.pl> wrote:

> > So the point is that you cannot remove the semaphore when there are no
> > callers waiting in semop because that is a perfectly valid state.
>
> Yes, but if function which acquires a semaphore is constructed as
> my_sem_acquire function it will be safe to remove the semaphore when it is
> released (if semaphore is "unlocked").

How would your code decide when to actually go ahead and remove the
semaphore?

Are you suggesting that the semaphore simply be removed whenever it
is determined to be "unlocked"? Meaning you want to call semget() and
semctl(IPC_RMID) for every single lock/unlock?

Mike

--
Michael B Allen
PHP Active Directory SSO
http://www.ioplex.com/

Wojciech Malota

19 years ago

> How would your code decide when to actually go ahead and remove the
> semaphore?
>
> Are you suggesting that the semaphore simply be removed whenever it
> is determined to be "unlocked"? Meaning you want to call semget() and
> semctl(IPC_RMID) for every single lock/unlock?

I've wrote about it in my first letter.
There are two functions: my_sem_acquire and my_sem_release.

Wojciech Malota

Michael B Allen

19 years ago

On Wed, 3 Jan 2007 06:51:57 +0100
"Wojciech Malota" <wojtekm86@konto.pl> wrote:

> > How would your code decide when to actually go ahead and remove the
> > semaphore?
> >
> > Are you suggesting that the semaphore simply be removed whenever it
> > is determined to be "unlocked"? Meaning you want to call semget() and
> > semctl(IPC_RMID) for every single lock/unlock?
>
> I've wrote about it in my first letter.
> There are two functions: my_sem_acquire and my_sem_release.

Well I read your original post again. Clearly I've been off track.

So the problem is that one process may call semop AFTER another process
calls semctl(IPC_RMID) to remove the semaphore from the system?

If that's the case then it is still impossible to know if it's ok to
remove a semaphore. Only the application would know when all paticipants
will no longer need the semaphore and that it can be safely removed. I'm
not familiar with the PHP semaphore API but I know a lot about the
SysV semphore C api and there's no "FIFO" that could be used to somehow
indicate that it's safe to remove a semaphore.

You must use some additional application specific logic to trigger
semaphore removal. For example you could have a "boss" process that
collects the exit status of each "worker". When the boss process has an
exit status for all workers the semaphore can be removed.

Am I getting closer or am I still not understanding the problem?

Mike

--
Michael B Allen
PHP Active Directory SSO
http://www.ioplex.com/

Wojciech Malota

19 years ago

> You must use some additional application specific logic to trigger
> semaphore removal. For example you could have a "boss" process that
> collects the exit status of each "worker". When the boss process has an
> exit status for all workers the semaphore can be removed.
>
> Am I getting closer or am I still not understanding the problem?
>
> Mike

You are much closer now.
I think that PHP interpreter should be a "boss" because PHP scripts don't
know if this execution of website is the last which releases the semaphore.
So if last script calls my_sem_release function, the semaphore will be
removed and now another script calls my_sem_acquire function it won't cause
an error because inside my_sem_acquire function is called sem_get() which
will create this semaphore once again. So this solution is completely safe.
I haven't got knowledge about PHP source code, so I can't implement this
behaviour - I give only an idea.
The key to success is that all instructions in functions my_sem_acquire and
my_sem_release can't be interrupted.

Wojciech Malota