Yes, PRISM does work with PHP 5.5.0, Alpha 1 without any modification, please test and let me know how you like it ... I would be nice to have a PRISM 0.5.0 that runs on PHP 5.5.0, I like the synergy of the fives.
PRISM & PHP 5.5.0 Alpha 1
(17 posts, started )
PRISM & PHP 5.5.0 Alpha 1
#2
- cargame.nl
Hmm well..
I've seen this pattern of release dates somewhere else.
I've seen this pattern of release dates somewhere else.
2015 that's optimistic, more like 2055, I like the synergy more.
Victor, take a look at the video that I linked to below for the network portion of PRISM 0.5.0 using PHP 5.5.0.
The video it's self is of the networking portion of BBQSQL, written in Python. They are using Coroutines (Now available in PHP 5.5.x) to get a lot of proformance in their networking IO because it switches from the first IO request to the next when it get's to an blocking IO call. Could be interesting for the next version of PRISM's networking stack.
https://www.youtube.com/watch?v=I4XRnuAA-aA#t=32m30s
The video it's self is of the networking portion of BBQSQL, written in Python. They are using Coroutines (Now available in PHP 5.5.x) to get a lot of proformance in their networking IO because it switches from the first IO request to the next when it get's to an blocking IO call. Could be interesting for the next version of PRISM's networking stack.
https://www.youtube.com/watch?v=I4XRnuAA-aA#t=32m30s
i have briefly looked at that portion of the video. Will take a closer look at coroutines soon - looks interesting.
But the networking part of prism is non blocking, so it should never wait. Except of course in the select call. That's the only place where, for network IO, the program will wait.
But at this point I realise i'll have to have a closer look at how coroutines really work (in practise and not just theory), in order say anything sensible about how they would or could help.
Will get php5.5a2 tomorrow and play a bit.
ps, don't the guys in the video refer to SQL queries as blocking IO? I think that's the part where a program will essentially wait the most. More than networking, which can be done asynchronous. But db queries are usually synchronous (blocking)
But the networking part of prism is non blocking, so it should never wait. Except of course in the select call. That's the only place where, for network IO, the program will wait.
But at this point I realise i'll have to have a closer look at how coroutines really work (in practise and not just theory), in order say anything sensible about how they would or could help.
Will get php5.5a2 tomorrow and play a bit.
ps, don't the guys in the video refer to SQL queries as blocking IO? I think that's the part where a program will essentially wait the most. More than networking, which can be done asynchronous. But db queries are usually synchronous (blocking)
Yeah, they are primarily talking about SQL query, they are the devs for BBQSQL, that basically allows you to quickly find out the information about the SQL server from various attack vectors available to it. Having a non-blocking SQL part to RRISM would be immensely helpful as well.
ok I've been reading more about generators and coroutines and I've watched more of that bbqsql video and I think it's starting to become clear what it's all about.
What the bbq guys use coroutines for is basically iterating over an array of sql queries, therby achieving the effect of multithreading.
Normally a query is blocking, and for good reason because the program cannot run without the data it has requested. It needs that data, so it has to wait for it to come from the db. For example a login query. The program cannot execute further instructions until it knows the status of the login attempt.
Sometimes though, like the bbq example, you want to just fire away a whole load of queries and the program doesn't rely on the answers. It doesn't matter if query 1 takes longer than query 2. All you care about is the end result, the collection of answers from all queries. So things are now allowed to run asynchronously. You could start a new thread for each query, or as bbq does, use coroutines with non blocking sql queries. This makes sense because the program will be waiting a relatively long time for the db to answer, so the cpu has time for other things while it waits.
So what bbq does is iterate over a collection of queries.
This does not translate 1:1 to prism. Prism (in its base) doesn't have iterations that use blocking IO. So I do not see any use for coroutines there.
The only blocking line of code is the select call and it should block as so not to waste cpu resources doing the main loop for no reason.
The only area where coroutines could be useful imo is plugins. But only if a plugin needs it. It doesn't make sense to turn every db query into a non blocking one using a coroutine.
If a plugin needs some information from a database - maybe the status of a racer who just joined - you could of course make that non blocking. You could make the plugin execute the query and return control to the main loop. But then the main loop needs to be notified that that particular plugin cannot be run again until the asynchronous query it's still waiting for has finished. It will complicate things a lot. And I don't think coroutines are made for this purpose.
They are made to make iterations easier. Parsing huge files for example, with a low memory footprint. Or doing many sql queries simultaneously to simulate threading. Processing of (big) collections is what generators and coroutines do best, as far as I understood it.
Generators and coroutines seem to be nice tools, but as with all tools they only work when used for the right job.
EDIT - add the usual disclaimers. I could be mistaken, as I haven't even tried them yet myself. So if you don't agree, discuss
What the bbq guys use coroutines for is basically iterating over an array of sql queries, therby achieving the effect of multithreading.
Normally a query is blocking, and for good reason because the program cannot run without the data it has requested. It needs that data, so it has to wait for it to come from the db. For example a login query. The program cannot execute further instructions until it knows the status of the login attempt.
Sometimes though, like the bbq example, you want to just fire away a whole load of queries and the program doesn't rely on the answers. It doesn't matter if query 1 takes longer than query 2. All you care about is the end result, the collection of answers from all queries. So things are now allowed to run asynchronously. You could start a new thread for each query, or as bbq does, use coroutines with non blocking sql queries. This makes sense because the program will be waiting a relatively long time for the db to answer, so the cpu has time for other things while it waits.
So what bbq does is iterate over a collection of queries.
This does not translate 1:1 to prism. Prism (in its base) doesn't have iterations that use blocking IO. So I do not see any use for coroutines there.
The only blocking line of code is the select call and it should block as so not to waste cpu resources doing the main loop for no reason.
The only area where coroutines could be useful imo is plugins. But only if a plugin needs it. It doesn't make sense to turn every db query into a non blocking one using a coroutine.
If a plugin needs some information from a database - maybe the status of a racer who just joined - you could of course make that non blocking. You could make the plugin execute the query and return control to the main loop. But then the main loop needs to be notified that that particular plugin cannot be run again until the asynchronous query it's still waiting for has finished. It will complicate things a lot. And I don't think coroutines are made for this purpose.
They are made to make iterations easier. Parsing huge files for example, with a low memory footprint. Or doing many sql queries simultaneously to simulate threading. Processing of (big) collections is what generators and coroutines do best, as far as I understood it.
Generators and coroutines seem to be nice tools, but as with all tools they only work when used for the right job.
EDIT - add the usual disclaimers. I could be mistaken, as I haven't even tried them yet myself. So if you don't agree, discuss
It's not that I disagree with you, I just think of it from an end programmers point of view and an end user point of view. From the programmers point of view there are some thing that I would like to know, but don't need to know right away. For example, a clients database information to update their UI state. I can wait for this information to become available. From a users point of view, If I can avoid a blocking IO function in an SQL query that provides me with a responsive user interface. That is massive! If we can pull that off with the tools given that would help the programmer and the user.
I see this happening in two ways. I've already discussed the first case above, where I don't need to know the details of the SQL query right, I can wait for these resources to be yielded to me. However this does not work if I am trying to authenticate a user. For that, I may need some purposefully blocking IO here to make sure that the next part of the function is only called in the case where my client is authenticated and authorized. I see the query prototype for this method looking something like this.
SQLResult SQL::query ( string $query [, bool $yield = TRUE]);
Using this method above, PRISM would create an array of query strings and callback functions. It would keep track of what query string should go to what function once it's result yields. It then iterates over this array at the end of the main loop. This should allow for blocking and non-blocking querys. The non-blocking queries should not block the main loop of PRISM, in theory at least, meaning it should allow for the continued processing of packets even while there are outstanding SQL queries.
Maybe I've read the spec wrong, maybe python's implamentation of coroutines is different then that of PHP's. maybe I can't yield a SQL query. But if I can, I think that it would be a huge win for the programmers and end users of PRISM. I would love to start working on a proof of concept for this, but it seems that I'm going to be stuck working for the rest of my life. But this would be very cool to see. Maybe if I get a night I make a build of PRISM with coroutines, maybe it will be awesome, maybe it won't.
I see this happening in two ways. I've already discussed the first case above, where I don't need to know the details of the SQL query right, I can wait for these resources to be yielded to me. However this does not work if I am trying to authenticate a user. For that, I may need some purposefully blocking IO here to make sure that the next part of the function is only called in the case where my client is authenticated and authorized. I see the query prototype for this method looking something like this.
SQLResult SQL::query ( string $query [, bool $yield = TRUE]);
Using this method above, PRISM would create an array of query strings and callback functions. It would keep track of what query string should go to what function once it's result yields. It then iterates over this array at the end of the main loop. This should allow for blocking and non-blocking querys. The non-blocking queries should not block the main loop of PRISM, in theory at least, meaning it should allow for the continued processing of packets even while there are outstanding SQL queries.
Maybe I've read the spec wrong, maybe python's implamentation of coroutines is different then that of PHP's. maybe I can't yield a SQL query. But if I can, I think that it would be a huge win for the programmers and end users of PRISM. I would love to start working on a proof of concept for this, but it seems that I'm going to be stuck working for the rest of my life. But this would be very cool to see. Maybe if I get a night I make a build of PRISM with coroutines, maybe it will be awesome, maybe it won't.
The first problem you'll run into is that prism uses PDO. That cannot do asynchronous (mysql) queries. Only mysqli with mysqlnd has it. So unless you want to change that or add a second DB handling class to prism, you can forget about all this right here.
Second, to just do asynchronous db queries, you don't need coroutines. You can just .. do it. Fire the query and at intervals poll to see if a result is back yet. But only with mysqli+mysqlnd. No sqlite asynch support.
Third, Coroutines came into play with the bbq guys because they wanted to iterate over a lot , I mean A LOT of queries. Coroutines didn't magically make things work faster or so. I have the feeling that's how you still see it.
Yes, it will be best if you just play with it when you have some time. Make a small test script to see how things actually work.
Second, to just do asynchronous db queries, you don't need coroutines. You can just .. do it. Fire the query and at intervals poll to see if a result is back yet. But only with mysqli+mysqlnd. No sqlite asynch support.
Third, Coroutines came into play with the bbq guys because they wanted to iterate over a lot , I mean A LOT of queries. Coroutines didn't magically make things work faster or so. I have the feeling that's how you still see it.
Yes, it will be best if you just play with it when you have some time. Make a small test script to see how things actually work.
This post is not directly related to the topic, but it is regarding async mysql queries:
I was already working on improving the speed of the charts window on lfsworld, and thinking about async queries I thought, hey I can try that on the charts and load the 4 charts simultaneously.
Turns out that's actually slower than doing them serially. That's probably because they were not super fast queries to start with so the db has some work to do, and now that workload quadrupled (although each thread should be able to run on a physical core) and also because you have to create a db connection for every concurrent query, which adds overhead.
So that led me to conclude it's really not that useful to use asynch queries to parallelize 'a handful of queries'. Whereas (again comparing to) the bbq guys they open a range of connections and then reuse them all the time. That's already better. And they use queries with small result sets, which I guess improves performance as well.
So what I'm saying here is that async queries work best in specific scenarios only. Even if you can really parallelize a number of queries, it doesn't mean it's actually faster. Or if you'd use async queries somehow to have non-blocking database traffic, then the end result could actually still mean more work for the cpu.
I really advise you think again about this for prism. It's not that useful. It's one of those things that sounds cool, but only has a very narrow field of application. It's much better to make sure you and people make proper db table layouts and queries. They don't have to be in the way of networking, provided they are written properly.
Of course when you start using giant datasets with millions of rows, then you may encounter problems if you want to do certain things with that data that it wasn't designed to do, but I kinda doubt anyone using prism will go that far.
I was already working on improving the speed of the charts window on lfsworld, and thinking about async queries I thought, hey I can try that on the charts and load the 4 charts simultaneously.
Turns out that's actually slower than doing them serially. That's probably because they were not super fast queries to start with so the db has some work to do, and now that workload quadrupled (although each thread should be able to run on a physical core) and also because you have to create a db connection for every concurrent query, which adds overhead.
So that led me to conclude it's really not that useful to use asynch queries to parallelize 'a handful of queries'. Whereas (again comparing to) the bbq guys they open a range of connections and then reuse them all the time. That's already better. And they use queries with small result sets, which I guess improves performance as well.
So what I'm saying here is that async queries work best in specific scenarios only. Even if you can really parallelize a number of queries, it doesn't mean it's actually faster. Or if you'd use async queries somehow to have non-blocking database traffic, then the end result could actually still mean more work for the cpu.
I really advise you think again about this for prism. It's not that useful. It's one of those things that sounds cool, but only has a very narrow field of application. It's much better to make sure you and people make proper db table layouts and queries. They don't have to be in the way of networking, provided they are written properly.
Of course when you start using giant datasets with millions of rows, then you may encounter problems if you want to do certain things with that data that it wasn't designed to do, but I kinda doubt anyone using prism will go that far.
I see your point. I think at this point we would need some hard data to prove that it would work better, like some bench mark results. But I really don't have any desire to add MySQL support to PRISM. I'm only happy with SQLite because it comes in the engine's core, so we can be sure that everyone will have it.
Are you sure?
The docs say :
And at least here on freebsd that is confirmed. The php 5.4 core does not contain sqlite. If I want pdo sqlite support I have to compile the modules php_pdo and php_pdo_sqlite. And if I'm not mistaken, that is sqlite3, not sqlite2 which was bundled with php prior to 5.4 .
EDIT - i agree with you though about sticking with sqlite. It's prefectly fine for prism. And imo it's no biggy if people need to add some modules for that to work.
sqlite3 is bundled with the core prior to 5.4, at least it was for me, when I compiled php with the default `config` and `make` settings. Ya know I did not notice that they changed that with 5.4, man the PHP's devs a freaking killing me here! Now we don't have a single 'core' db engine that we can count on being there ... ****!
Yeah, I think it's the best fit for PRISM. The fact of the matter is right now, you need to have the sockets module installed for PRISM to function at all, so I feel that having this also is a no-brainier, just like having the sockets module.
Looks like the PHP Devs are getting rid of ext/mysql in the next versions of PHP. PHP 5.5.0 Alpha 2 deprecates it, with full on removal expected in PHP 5.6.x and PHP 6.0.
https://wiki.php.net/rfc/mysql_deprecation
They will keep PDO support and MySQLi support as PDO is "the PHP Way" and MySQLi supports async queries.
Still only MySQLi supports async queries tho.
http://blog.ulf-wendel.de/2008 ... s-help-you-with-sharding/
http://planet.mysql.com/entry/?id=32136
https://wiki.php.net/rfc/mysql_deprecation
They will keep PDO support and MySQLi support as PDO is "the PHP Way" and MySQLi supports async queries.
Still only MySQLi supports async queries tho.
http://blog.ulf-wendel.de/2008 ... s-help-you-with-sharding/
http://planet.mysql.com/entry/?id=32136
I don't think that'll be the last word on it. I'm surprised they want to drop the mysql module. Things like Mediawiki still use it exclusively, to this day! For some reason they don't want to implement mysqli support...
So that is .. weird. I'll be surprised if they remove it in 6.
I think they will drop it in 5.6 / 6.0 compleatly, from the threads I've been reading in the PHP mailing list, they hate maintaining support for it, it's a headache to keep it going ... kinda like putting a band aid on a zombie, the underlying problem is still that it's a zombie. That's kinda how they feel about it. So at this point forward, it's PDO or MySQLi for MySQL support within PHP.
So, part of the rewrite that I'm doing for PHP 5.5, is the InSim, HTTP and Telnet interfaces are no longer going to be hard coded into the core of PRISM. I want to be able to implement any type of interface that uses sockets, simply by importing it with a configuration file.
This means a few things, for the core devs. Each interface has to promise to provide a checkTraffic() method and a getSelectableSockets() method. It can also implement a maintenance() method if the interface requires it (InSim Did).
The cool part about doing it this way, is that PRISM becomes quite a bit more general purpose and can be suited to interact with many different types of interfaces quite easily and because of the plugin system attached to it, it can cross pollinate the information from one interface (InSim for Example) with another (WebSockets, Telnet, Ect).
There is another thing, that I wanted to discuss. The deprecation of the HTTP module. I'd like to use the built in PHP server as it should in theory make it easier for people to program for ... if we give them some boilerplate WebSocket interface for them to use. Basically, I wanted to replace the HTTP module with a WebSocket module, and fork anything that we did for HTTP and move that into a separate (but equal) project.
This means a few things, for the core devs. Each interface has to promise to provide a checkTraffic() method and a getSelectableSockets() method. It can also implement a maintenance() method if the interface requires it (InSim Did).
The cool part about doing it this way, is that PRISM becomes quite a bit more general purpose and can be suited to interact with many different types of interfaces quite easily and because of the plugin system attached to it, it can cross pollinate the information from one interface (InSim for Example) with another (WebSockets, Telnet, Ect).
There is another thing, that I wanted to discuss. The deprecation of the HTTP module. I'd like to use the built in PHP server as it should in theory make it easier for people to program for ... if we give them some boilerplate WebSocket interface for them to use. Basically, I wanted to replace the HTTP module with a WebSocket module, and fork anything that we did for HTTP and move that into a separate (but equal) project.
PRISM & PHP 5.5.0 Alpha 1
(17 posts, started )
FGED GREDG RDFGDR GSFDG